Kubernetes Operators Deep Dive

March 26, 2026 18 min read Virtualization & Containers

kubernetesoperatorscontroller-runtimegoplatform-engineeringdevopscrd

Contents

Kubernetes operators are the mechanism by which Kubernetes becomes a platform for anything, not just containers. An operator is a controller that extends the Kubernetes API with custom resource types and encodes the operational knowledge for running a specific piece of software — the same knowledge a human operator would use, but expressed as code.

The Operator pattern is powerful but often poorly implemented in practice. Most tutorials show a toy reconciler that prints a log line. Real operators need to handle partial failure, track status accurately, clean up resources on deletion, run exactly one reconciler at a time in a cluster, and be testable without a running cluster. This guide covers all of it.

The Control Loop Mental Model

Before writing code, internalize the control loop:

Desired state (what you wrote in YAML) ──→ Controller ──→ Actual state (what's running)
                                               ↑
                                    Observe → Compare → Act

A controller watches resources, compares observed state to desired state, and takes actions to reconcile the difference. Crucially:

Reconciliation is level-triggered, not edge-triggered. You don’t respond to “this changed” events — you respond to “here is the current state, make the world match it.” If your controller crashes mid-reconcile and restarts, it re-reads the current state and continues. Idempotency is mandatory.
Reconciliation is eventually consistent. Your controller may be called many times before reaching the desired state. Each call should make progress or return an error to retry.
You don’t own the world. Other controllers, users, and systems change resources underneath you. Re-read current state at the start of every reconcile; never assume your last-written values are still there.

Project Setup

Use kubebuilder — the standard scaffolding tool for operator projects:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


# Install kubebuilder
curl -L -o kubebuilder "https://go.kubebuilder.io/dl/latest/$(go env GOOS)/$(go env GOARCH)"
chmod +x kubebuilder && sudo mv kubebuilder /usr/local/bin/

# Create a new operator project
mkdir webservice-operator && cd webservice-operator
kubebuilder init \
  --domain lunarops.io \
  --repo github.com/lunarops/webservice-operator

# Scaffold a new API (CRD + Controller)
kubebuilder create api \
  --group platform \
  --version v1 \
  --kind WebService \
  --resource \
  --controller

This generates:

webservice-operator/
├── api/
│   └── v1/
│       ├── webservice_types.go      # CRD type definitions
│       └── groupversion_info.go
├── internal/
│   └── controller/
│       ├── webservice_controller.go # reconcile logic
│       └── webservice_controller_test.go
├── config/
│   ├── crd/                         # CRD manifests (generated)
│   ├── rbac/                        # ClusterRole manifests (generated)
│   └── manager/                     # operator Deployment manifest
├── cmd/
│   └── main.go                      # entrypoint
└── Makefile

CRD Design

The CRD is your API contract. Design it carefully — changing it later is painful.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111


// api/v1/webservice_types.go

package v1

import (
    corev1 "k8s.io/api/core/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// WebServiceSpec defines the desired state
type WebServiceSpec struct {
    // +kubebuilder:validation:Required
    // +kubebuilder:validation:MinLength=1
    Image string `json:"image"`

    // +kubebuilder:default=8080
    // +kubebuilder:validation:Minimum=1
    // +kubebuilder:validation:Maximum=65535
    Port int32 `json:"port,omitempty"`

    // +kubebuilder:validation:Minimum=1
    // +kubebuilder:validation:Maximum=100
    // +kubebuilder:default=2
    Replicas *int32 `json:"replicas,omitempty"`

    // Resource requirements for the container
    Resources corev1.ResourceRequirements `json:"resources,omitempty"`

    // Environment variables injected into the container
    Env []corev1.EnvVar `json:"env,omitempty"`

    // Ingress configuration — nil means no ingress
    Ingress *IngressSpec `json:"ingress,omitempty"`

    // Database configuration — nil means no database
    Database *DatabaseSpec `json:"database,omitempty"`
}

type IngressSpec struct {
    // +kubebuilder:validation:Required
    Host string `json:"host"`

    // +kubebuilder:default=true
    TLS bool `json:"tls,omitempty"`
}

type DatabaseSpec struct {
    // +kubebuilder:validation:Enum=postgres;mysql
    // +kubebuilder:default=postgres
    Type string `json:"type,omitempty"`

    // +kubebuilder:validation:Enum=small;medium;large
    // +kubebuilder:default=small
    Size string `json:"size,omitempty"`
}

// WebServiceStatus defines the observed state
type WebServiceStatus struct {
    // Conditions represent the latest available observations of the object's state
    // +patchMergeKey=type
    // +patchStrategy=merge
    // +listType=map
    // +listMapKey=type
    Conditions []metav1.Condition `json:"conditions,omitempty"`

    // ReadyReplicas is the number of pods currently ready
    ReadyReplicas int32 `json:"readyReplicas,omitempty"`

    // ObservedGeneration is the .metadata.generation this status was computed from
    ObservedGeneration int64 `json:"observedGeneration,omitempty"`

    // Phase summarises the overall state
    // +kubebuilder:validation:Enum=Pending;Deploying;Ready;Degraded;Failed
    Phase string `json:"phase,omitempty"`
}

// Condition type constants
const (
    // ConditionDeploymentReady indicates the Deployment is ready
    ConditionDeploymentReady = "DeploymentReady"
    // ConditionIngressReady indicates the Ingress has been assigned an address
    ConditionIngressReady = "IngressReady"
    // ConditionDatabaseReady indicates the database has been provisioned
    ConditionDatabaseReady = "DatabaseReady"
)

//+kubebuilder:object:root=true
//+kubebuilder:subresource:status
//+kubebuilder:printcolumn:name="Phase",type="string",JSONPath=".status.phase"
//+kubebuilder:printcolumn:name="Ready",type="integer",JSONPath=".status.readyReplicas"
//+kubebuilder:printcolumn:name="Age",type="date",JSONPath=".metadata.creationTimestamp"

type WebService struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec   WebServiceSpec   `json:"spec,omitempty"`
    Status WebServiceStatus `json:"status,omitempty"`
}

//+kubebuilder:object:root=true

type WebServiceList struct {
    metav1.TypeMeta `json:",inline"`
    metav1.ListMeta `json:"metadata,omitempty"`
    Items           []WebService `json:"items"`
}

func init() {
    SchemeBuilder.Register(&WebService{}, &WebServiceList{})
}

Generate CRD manifests and deepcopy methods after changes:

1
2


make generate    # generates DeepCopyObject methods
make manifests   # generates CRD YAML in config/crd/

The Reconciler

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120


// internal/controller/webservice_controller.go

package controller

import (
    "context"
    "fmt"

    appsv1 "k8s.io/api/apps/v1"
    corev1 "k8s.io/api/core/v1"
    networkingv1 "k8s.io/api/networking/v1"
    "k8s.io/apimachinery/pkg/api/errors"
    "k8s.io/apimachinery/pkg/api/meta"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/apimachinery/pkg/runtime"
    "k8s.io/apimachinery/pkg/util/intstr"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"
    "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
    "sigs.k8s.io/controller-runtime/pkg/log"

    platformv1 "github.com/lunarops/webservice-operator/api/v1"
)

const (
    finalizerName = "platform.lunarops.io/finalizer"
)

type WebServiceReconciler struct {
    client.Client
    Scheme *runtime.Scheme
}

// +kubebuilder:rbac:groups=platform.lunarops.io,resources=webservices,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=platform.lunarops.io,resources=webservices/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=platform.lunarops.io,resources=webservices/finalizers,verbs=update
// +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=core,resources=services,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=networking.k8s.io,resources=ingresses,verbs=get;list;watch;create;update;patch;delete

func (r *WebServiceReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    logger := log.FromContext(ctx)

    // 1. Fetch the WebService resource
    ws := &platformv1.WebService{}
    if err := r.Get(ctx, req.NamespacedName, ws); err != nil {
        if errors.IsNotFound(err) {
            // Object deleted between event and reconcile — nothing to do
            return ctrl.Result{}, nil
        }
        return ctrl.Result{}, fmt.Errorf("failed to get WebService: %w", err)
    }

    // 2. Handle deletion with a finalizer
    if !ws.DeletionTimestamp.IsZero() {
        return r.handleDeletion(ctx, ws)
    }

    // 3. Add finalizer if not present
    if !controllerutil.ContainsFinalizer(ws, finalizerName) {
        controllerutil.AddFinalizer(ws, finalizerName)
        if err := r.Update(ctx, ws); err != nil {
            return ctrl.Result{}, fmt.Errorf("failed to add finalizer: %w", err)
        }
        // Return and re-queue — the Update will trigger another reconcile
        return ctrl.Result{}, nil
    }

    // 4. Update ObservedGeneration to track spec changes
    if ws.Status.ObservedGeneration != ws.Generation {
        ws.Status.ObservedGeneration = ws.Generation
        ws.Status.Phase = "Deploying"
    }

    // 5. Reconcile each sub-resource
    deploymentReady, err := r.reconcileDeployment(ctx, ws)
    if err != nil {
        r.setCondition(ws, ConditionDeploymentReady, metav1.ConditionFalse,
            "ReconcileError", err.Error())
        _ = r.updateStatus(ctx, ws)
        return ctrl.Result{}, err
    }

    serviceReady, err := r.reconcileService(ctx, ws)
    if err != nil {
        return ctrl.Result{}, err
    }

    ingressReady := true
    if ws.Spec.Ingress != nil {
        ingressReady, err = r.reconcileIngress(ctx, ws)
        if err != nil {
            return ctrl.Result{}, err
        }
    }

    // 6. Update status conditions and phase
    r.setCondition(ws, ConditionDeploymentReady,
        boolToConditionStatus(deploymentReady), "Reconciled", "")
    r.setCondition(ws, ConditionIngressReady,
        boolToConditionStatus(ingressReady && serviceReady), "Reconciled", "")

    if deploymentReady && serviceReady && ingressReady {
        ws.Status.Phase = "Ready"
    } else {
        ws.Status.Phase = "Deploying"
    }

    if err := r.updateStatus(ctx, ws); err != nil {
        return ctrl.Result{}, err
    }

    logger.Info("Reconciled WebService",
        "name", ws.Name,
        "phase", ws.Status.Phase,
        "readyReplicas", ws.Status.ReadyReplicas)

    // Re-queue after 30 seconds to catch drift
    return ctrl.Result{RequeueAfter: 30 * time.Second}, nil
}

Reconciling a Deployment with CreateOrUpdate

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84


func (r *WebServiceReconciler) reconcileDeployment(
    ctx context.Context, ws *platformv1.WebService,
) (bool, error) {

    replicas := int32(2)
    if ws.Spec.Replicas != nil {
        replicas = *ws.Spec.Replicas
    }

    deploy := &appsv1.Deployment{
        ObjectMeta: metav1.ObjectMeta{
            Name:      ws.Name,
            Namespace: ws.Namespace,
        },
    }

    // CreateOrUpdate: fetch existing, apply mutations, create/update
    result, err := controllerutil.CreateOrUpdate(ctx, r.Client, deploy, func() error {
        // Set owner reference — Deployment is garbage-collected when WebService is deleted
        if err := controllerutil.SetControllerReference(ws, deploy, r.Scheme); err != nil {
            return err
        }

        // Labels for selection
        labels := map[string]string{
            "app.kubernetes.io/name":       ws.Name,
            "app.kubernetes.io/managed-by": "webservice-operator",
        }

        deploy.Spec = appsv1.DeploymentSpec{
            Replicas: &replicas,
            Selector: &metav1.LabelSelector{
                MatchLabels: labels,
            },
            Template: corev1.PodTemplateSpec{
                ObjectMeta: metav1.ObjectMeta{
                    Labels: labels,
                },
                Spec: corev1.PodSpec{
                    SecurityContext: &corev1.PodSecurityContext{
                        RunAsNonRoot: ptr(true),
                    },
                    Containers: []corev1.Container{
                        {
                            Name:  ws.Name,
                            Image: ws.Spec.Image,
                            Ports: []corev1.ContainerPort{
                                {ContainerPort: ws.Spec.Port, Protocol: corev1.ProtocolTCP},
                            },
                            Env: ws.Spec.Env,
                            Resources: ws.Spec.Resources,
                            SecurityContext: &corev1.SecurityContext{
                                AllowPrivilegeEscalation: ptr(false),
                                ReadOnlyRootFilesystem:   ptr(true),
                            },
                            ReadinessProbe: &corev1.Probe{
                                ProbeHandler: corev1.ProbeHandler{
                                    HTTPGet: &corev1.HTTPGetAction{
                                        Path: "/healthz",
                                        Port: intstr.FromInt32(ws.Spec.Port),
                                    },
                                },
                                InitialDelaySeconds: 5,
                                PeriodSeconds:       10,
                            },
                        },
                    },
                },
            },
        }
        return nil
    })

    if err != nil {
        return false, fmt.Errorf("failed to reconcile Deployment: %w", err)
    }
    if result != controllerutil.OperationResultNone {
        log.FromContext(ctx).Info("Deployment reconciled", "operation", result)
    }

    // Check if the deployment is ready
    ws.Status.ReadyReplicas = deploy.Status.ReadyReplicas
    return deploy.Status.ReadyReplicas == replicas, nil
}

Finalizers: Cleanup on Deletion

Without a finalizer, when a user deletes a WebService, Kubernetes immediately removes it from the API. Your controller never gets a chance to clean up external resources — provisioned databases, DNS records, certificates. A finalizer prevents this.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30


func (r *WebServiceReconciler) handleDeletion(
    ctx context.Context, ws *platformv1.WebService,
) (ctrl.Result, error) {
    logger := log.FromContext(ctx)

    if !controllerutil.ContainsFinalizer(ws, finalizerName) {
        return ctrl.Result{}, nil
    }

    logger.Info("Running cleanup for deleted WebService", "name", ws.Name)

    // Perform external cleanup
    if ws.Spec.Database != nil {
        if err := r.deleteDatabase(ctx, ws); err != nil {
            // Cleanup failed — update status and retry
            ws.Status.Phase = "Failed"
            _ = r.updateStatus(ctx, ws)
            return ctrl.Result{}, fmt.Errorf("failed to delete database: %w", err)
        }
    }

    // Remove the finalizer — Kubernetes will then garbage-collect the resource
    controllerutil.RemoveFinalizer(ws, finalizerName)
    if err := r.Update(ctx, ws); err != nil {
        return ctrl.Result{}, fmt.Errorf("failed to remove finalizer: %w", err)
    }

    logger.Info("Finalizer removed, WebService will be deleted", "name", ws.Name)
    return ctrl.Result{}, nil
}

Finalizer pitfalls:

If handleDeletion always errors, the object is stuck — never deleted. Always provide a way to force-remove the finalizer for recovery: kubectl patch webservice myapp --type=json -p='[{"op":"remove","path":"/metadata/finalizers"}]'
Keep cleanup idempotent — it may be called multiple times if the controller restarts during deletion.
Don’t add finalizers to objects you don’t own. Only add them to your own custom resources.

Status Conditions: Accurate Observability

Status conditions are the standard way to report the health of a Kubernetes object. They’re typed, timestamped, and composable — tools like kubectl wait and GitOps controllers use them.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


func (r *WebServiceReconciler) setCondition(
    ws *platformv1.WebService,
    conditionType string,
    status metav1.ConditionStatus,
    reason, message string,
) {
    now := metav1.Now()
    condition := metav1.Condition{
        Type:               conditionType,
        Status:             status,
        ObservedGeneration: ws.Generation,
        LastTransitionTime: now,
        Reason:             reason,
        Message:            message,
    }
    // Only update LastTransitionTime if status actually changed
    meta.SetStatusCondition(&ws.Status.Conditions, condition)
}

func (r *WebServiceReconciler) updateStatus(
    ctx context.Context, ws *platformv1.WebService,
) error {
    // Always use StatusClient.Update for status — never r.Update
    // This prevents the spec from being overwritten accidentally
    if err := r.Status().Update(ctx, ws); err != nil {
        if errors.IsConflict(err) {
            // Another process updated the resource — re-queue
            return nil
        }
        return fmt.Errorf("failed to update status: %w", err)
    }
    return nil
}

Users can then check status with:

1
2
3
4
5
6
7


kubectl get webservice myapp -o jsonpath='{.status.conditions}'
kubectl wait webservice/myapp --for=condition=DeploymentReady --timeout=120s

# Custom columns from kubebuilder printcolumn markers
kubectl get webservice
# NAME    PHASE   READY   AGE
# myapp   Ready   3       5m

Watching Secondary Resources

Your controller needs to reconcile when resources it manages change — not just when the WebService itself changes.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42


func (r *WebServiceReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&platformv1.WebService{}).           // primary resource
        Owns(&appsv1.Deployment{}).              // re-queue WebService when owned Deployment changes
        Owns(&corev1.Service{}).
        Owns(&networkingv1.Ingress{}).
        // Watch unowned resources (e.g., a ConfigMap that affects many WebServices)
        Watches(
            &corev1.ConfigMap{},
            handler.EnqueueRequestsFromMapFunc(r.findWebServicesForConfigMap),
        ).
        // Limit concurrency — how many WebServices reconcile in parallel
        WithOptions(controller.Options{MaxConcurrentReconciles: 5}).
        Complete(r)
}

// Map a ConfigMap to all WebServices that reference it
func (r *WebServiceReconciler) findWebServicesForConfigMap(
    ctx context.Context, obj client.Object,
) []reconcile.Request {
    wsList := &platformv1.WebServiceList{}
    if err := r.List(ctx, wsList, client.InNamespace(obj.GetNamespace())); err != nil {
        return nil
    }

    var requests []reconcile.Request
    for _, ws := range wsList.Items {
        // Only re-queue if this WebService references this ConfigMap
        for _, envFrom := range ws.Spec.EnvFrom {
            if envFrom.ConfigMapRef != nil &&
                envFrom.ConfigMapRef.Name == obj.GetName() {
                requests = append(requests, reconcile.Request{
                    NamespacedName: types.NamespacedName{
                        Name:      ws.Name,
                        Namespace: ws.Namespace,
                    },
                })
            }
        }
    }
    return requests
}

Error Handling and Requeue Strategy

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


// Return strategies from Reconcile:

// 1. Success — re-queue after 30s to catch drift
return ctrl.Result{RequeueAfter: 30 * time.Second}, nil

// 2. Permanent error — log it, don't retry automatically
// (use for validation errors, invalid spec, etc.)
return ctrl.Result{}, nil   // no error = no automatic retry

// 3. Transient error — retry with exponential backoff
return ctrl.Result{}, fmt.Errorf("failed to create service: %w", err)
// controller-runtime will retry with backoff

// 4. Conflict — resource was modified, re-queue immediately
if errors.IsConflict(err) {
    return ctrl.Result{Requeue: true}, nil
}

// 5. Rate limiting for specific errors
return ctrl.Result{RequeueAfter: 5 * time.Second}, nil

Wrap errors with context so logs are useful:

1
2
3
4


if err := r.Create(ctx, deploy); err != nil {
    return ctrl.Result{}, fmt.Errorf("creating Deployment %s/%s: %w",
        deploy.Namespace, deploy.Name, err)
}

Leader Election

In production, you run multiple replicas of your operator for availability. But multiple controllers reconciling the same resource in parallel causes conflicts. Leader election ensures only one controller is active at a time:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


// cmd/main.go
func main() {
    mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
        Scheme:                 scheme,
        MetricsBindAddress:     ":8080",
        HealthProbeBindAddress: ":8081",
        LeaderElection:         true,              // enable leader election
        LeaderElectionID:       "webservice-operator.lunarops.io",
        // The leader holds a lease; others wait.
        // On leader crash, a new leader is elected after LeaseDuration.
        LeaseDuration: &[]time.Duration{15 * time.Second}[0],
        RenewDeadline: &[]time.Duration{10 * time.Second}[0],
        RetryPeriod:   &[]time.Duration{2 * time.Second}[0],
    })
    // ...
}

1
2
3


# RBAC for leader election (generated by kubebuilder markers)
# +kubebuilder:rbac:groups=coordination.k8s.io,resources=leases,verbs=get;list;watch;create;update;patch;delete
# +kubebuilder:rbac:groups="",resources=events,verbs=create;patch

Testing: Envtest + Ginkgo

The standard testing approach uses envtest — a real API server and etcd running in the test process, without a full cluster:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75


// internal/controller/suite_test.go

package controller_test

import (
    "context"
    "path/filepath"
    "testing"

    . "github.com/onsi/ginkgo/v2"
    . "github.com/onsi/gomega"
    "k8s.io/client-go/kubernetes/scheme"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"
    "sigs.k8s.io/controller-runtime/pkg/envtest"
    logf "sigs.k8s.io/controller-runtime/pkg/log"
    "sigs.k8s.io/controller-runtime/pkg/log/zap"

    platformv1 "github.com/lunarops/webservice-operator/api/v1"
    //+kubebuilder:scaffold:imports
)

var (
    k8sClient client.Client
    testEnv   *envtest.Environment
    ctx       context.Context
    cancel    context.CancelFunc
)

func TestControllers(t *testing.T) {
    RegisterFailHandler(Fail)
    RunSpecs(t, "Controller Suite")
}

var _ = BeforeSuite(func() {
    logf.SetLogger(zap.New(zap.WriteTo(GinkgoWriter), zap.UseDevMode(true)))
    ctx, cancel = context.WithCancel(context.TODO())

    testEnv = &envtest.Environment{
        CRDDirectoryPaths: []string{
            filepath.Join("..", "..", "config", "crd", "bases"),
        },
        ErrorIfCRDPathMissing: true,
    }

    cfg, err := testEnv.Start()
    Expect(err).NotTo(HaveOccurred())

    err = platformv1.AddToScheme(scheme.Scheme)
    Expect(err).NotTo(HaveOccurred())

    k8sClient, err = client.New(cfg, client.Options{Scheme: scheme.Scheme})
    Expect(err).NotTo(HaveOccurred())

    // Start the controller manager
    mgr, err := ctrl.NewManager(cfg, ctrl.Options{Scheme: scheme.Scheme})
    Expect(err).NotTo(HaveOccurred())

    err = (&WebServiceReconciler{
        Client: mgr.GetClient(),
        Scheme: mgr.GetScheme(),
    }).SetupWithManager(mgr)
    Expect(err).NotTo(HaveOccurred())

    go func() {
        defer GinkgoRecover()
        err = mgr.Start(ctx)
        Expect(err).NotTo(HaveOccurred(), "failed to run manager")
    }()
})

var _ = AfterSuite(func() {
    cancel()
    Expect(testEnv.Stop()).To(Succeed())
})

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102


// internal/controller/webservice_controller_test.go

var _ = Describe("WebService Controller", func() {
    const (
        timeout  = time.Second * 30
        interval = time.Millisecond * 250
    )

    Context("When creating a WebService", func() {
        It("Should create a Deployment and Service", func() {
            ws := &platformv1.WebService{
                ObjectMeta: metav1.ObjectMeta{
                    Name:      "test-service",
                    Namespace: "default",
                },
                Spec: platformv1.WebServiceSpec{
                    Image:    "nginx:alpine",
                    Port:     80,
                    Replicas: ptr(int32(2)),
                },
            }

            Expect(k8sClient.Create(ctx, ws)).To(Succeed())

            // Wait for the Deployment to be created
            deployKey := types.NamespacedName{Name: "test-service", Namespace: "default"}
            createdDeploy := &appsv1.Deployment{}
            Eventually(func() error {
                return k8sClient.Get(ctx, deployKey, createdDeploy)
            }, timeout, interval).Should(Succeed())

            // Verify Deployment spec
            Expect(createdDeploy.Spec.Template.Spec.Containers[0].Image).
                To(Equal("nginx:alpine"))
            Expect(*createdDeploy.Spec.Replicas).To(Equal(int32(2)))

            // Verify owner reference set correctly
            Expect(createdDeploy.OwnerReferences).To(HaveLen(1))
            Expect(createdDeploy.OwnerReferences[0].Name).To(Equal("test-service"))
            Expect(createdDeploy.OwnerReferences[0].Controller).To(Equal(ptr(true)))
        })

        It("Should set status conditions", func() {
            wsKey := types.NamespacedName{Name: "test-service", Namespace: "default"}
            ws := &platformv1.WebService{}

            Eventually(func() string {
                if err := k8sClient.Get(ctx, wsKey, ws); err != nil {
                    return ""
                }
                cond := meta.FindStatusCondition(ws.Status.Conditions,
                    ConditionDeploymentReady)
                if cond == nil {
                    return ""
                }
                return string(cond.Status)
            }, timeout, interval).Should(Equal(string(metav1.ConditionTrue)))
        })

        It("Should clean up resources when deleted", func() {
            ws := &platformv1.WebService{}
            Expect(k8sClient.Get(ctx, types.NamespacedName{
                Name:      "test-service",
                Namespace: "default",
            }, ws)).To(Succeed())

            Expect(k8sClient.Delete(ctx, ws)).To(Succeed())

            // Deployment should be garbage-collected (via ownerReference)
            deploy := &appsv1.Deployment{}
            Eventually(func() bool {
                err := k8sClient.Get(ctx, types.NamespacedName{
                    Name:      "test-service",
                    Namespace: "default",
                }, deploy)
                return errors.IsNotFound(err)
            }, timeout, interval).Should(BeTrue())
        })
    })

    Context("When updating a WebService image", func() {
        It("Should update the Deployment image", func() {
            wsKey := types.NamespacedName{Name: "test-service", Namespace: "default"}
            ws := &platformv1.WebService{}
            Expect(k8sClient.Get(ctx, wsKey, ws)).To(Succeed())

            ws.Spec.Image = "nginx:latest"
            Expect(k8sClient.Update(ctx, ws)).To(Succeed())

            deploy := &appsv1.Deployment{}
            Eventually(func() string {
                if err := k8sClient.Get(ctx, wsKey, deploy); err != nil {
                    return ""
                }
                if len(deploy.Spec.Template.Spec.Containers) == 0 {
                    return ""
                }
                return deploy.Spec.Template.Spec.Containers[0].Image
            }, timeout, interval).Should(Equal("nginx:latest"))
        })
    })
})

Run tests:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# Download envtest binaries (first time)
make envtest
export KUBEBUILDER_ASSETS=$(./bin/setup-envtest use --print path)

# Run tests
go test ./internal/controller/... -v

# With coverage
go test ./... -coverprofile=coverage.out
go tool cover -html=coverage.out

Deploying the Operator

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


# Build and push the image
make docker-build docker-push IMG=ghcr.io/lunarops/webservice-operator:v0.1.0

# Install CRDs
make install

# Deploy the operator
make deploy IMG=ghcr.io/lunarops/webservice-operator:v0.1.0

# Or generate a bundle and deploy with OLM (Operator Lifecycle Manager)
make bundle IMG=ghcr.io/lunarops/webservice-operator:v0.1.0
make bundle-build bundle-push BUNDLE_IMG=ghcr.io/lunarops/webservice-operator-bundle:v0.1.0

Example WebService to deploy:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


apiVersion: platform.lunarops.io/v1
kind: WebService
metadata:
  name: my-api
  namespace: production
spec:
  image: ghcr.io/lunarops/my-api:v1.2.3
  port: 8080
  replicas: 3
  resources:
    requests:
      cpu: 200m
      memory: 256Mi
    limits:
      cpu: 500m
      memory: 512Mi
  ingress:
    host: my-api.lunarops.io
    tls: true
  env:
    - name: LOG_LEVEL
      value: info

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


kubectl apply -f webservice.yaml
kubectl get webservice my-api
# NAME     PHASE   READY   AGE
# my-api   Ready   3       2m

kubectl describe webservice my-api
# Status:
#   Conditions:
#     Last Transition Time: 2026-03-26T10:00:00Z
#     Reason: Reconciled
#     Status: True
#     Type: DeploymentReady
#   Observed Generation: 1
#   Phase: Ready
#   Ready Replicas: 3

Common Pitfalls

Mutating the object you fetched without re-reading. Always re-fetch before updating, or use r.Update with the object you just r.Get’d. Stale objects cause Conflict errors.

Updating status with r.Update instead of r.Status().Update. r.Update sends the whole object — if someone else changed the spec between your Get and Update, you’ll overwrite their change. r.Status().Update only updates the status subresource.

Not setting owner references. Without owner references, child resources (Deployments, Services) outlive the parent custom resource. Use controllerutil.SetControllerReference.

Infinite reconcile loops. If your reconciler updates the object’s spec or metadata (not just status), it triggers another reconcile, which updates the object, which triggers another reconcile. Only write to .Status from inside the reconcile loop.

Blocking the reconcile goroutine. Don’t do long-running I/O directly in Reconcile — use goroutines or break work into smaller reconcile steps. A blocked reconciler starves other objects in the same controller.

Not handling errors.IsNotFound on sub-resources. The Deployment you created may have been deleted externally. Always handle 404 gracefully — recreate it rather than returning an error.

Operators are one of the most powerful ways to extend Kubernetes. Done well, they turn complex operational procedures into declarative YAML that any engineer on the team can use without understanding the underlying complexity. Done poorly, they’re a source of subtle bugs that are hard to debug under production conditions. The patterns here — level-triggered reconciliation, owner references, finalizers, status conditions, envtest — are the difference between a toy operator and one you can confidently run in production.