|
| 1 | +# Strict Field Validation |
| 2 | + |
| 3 | +## Overview |
| 4 | + |
| 5 | +By default, when your controller writes an object that contains fields not defined in the CRD schema, the API server: |
| 6 | + |
| 7 | +- Accepts the request |
| 8 | +- Drops the unknown fields |
| 9 | +- May only log a warning |
| 10 | + |
| 11 | +This can hide bugs and version skew between: |
| 12 | +- The controller code (Go types) and |
| 13 | +- The CRD schema installed in the cluster |
| 14 | + |
| 15 | +`controller-runtime` exposes [client.WithFieldValidation][client-field-validation-docs] to turn on strict server-side field validation for all client writes. When enabled, the API server returns a hard error instead of silently dropping unknown fields. |
| 16 | + |
| 17 | +CRDs should be installed before controllers. However, during upgrades this can be best effort rather than guaranteed. Deployment tools may apply resources without strict ordering, and even tools with ordering features don't guarantee the CRD update succeeds. For controllers using external CRDs from third-party operators, ordering may not be controllable at all. |
| 18 | + |
| 19 | +When controllers and CRDs get out of sync, controller writes may fail with unknown field errors, status updates may not persist, and conversion webhooks can fail if the CRD schema is outdated. Controllers may crash-loop until CRDs are updated. |
| 20 | + |
| 21 | +## What does it solve |
| 22 | + |
| 23 | +Strict validation prevents silent failures when your controller code and CRD schemas get out of sync. |
| 24 | + |
| 25 | +For example, you add a new field `status.newField` to your controller, but the CRD in the cluster hasn't been updated yet. When the controller calls `client.Status().Patch(...)`: |
| 26 | + |
| 27 | +**Without strict validation:** |
| 28 | +- API server drops `status.newField` silently |
| 29 | +- Controller sees no error |
| 30 | +- Field never appears on the object - confusing debugging |
| 31 | + |
| 32 | +**With strict validation:** |
| 33 | +- API server returns clear error |
| 34 | +- Controller knows CRDs need updating |
| 35 | +- Fails fast instead of silent data loss |
| 36 | + |
| 37 | +## When to use it |
| 38 | + |
| 39 | +Strict validation is a good fit when: |
| 40 | + |
| 41 | +- You own both the CRDs and the controllers |
| 42 | +- Your upgrade process applies CRDs first or ensures they update together |
| 43 | +- You want to fail fast when a controller writes fields not in the schema |
| 44 | +- You want to catch bugs in your types or conversions early |
| 45 | +- You use typed schemas, or explicitly mark dynamic data with `x-kubernetes-preserve-unknown-fields: true` |
| 46 | + |
| 47 | +## When NOT to use it |
| 48 | + |
| 49 | +Avoid strict validation in production when: |
| 50 | + |
| 51 | +- Controllers and CRDs upgrade independently (i.e., common in Helm) |
| 52 | +- You manage third-party CRDs whose schemas evolve independently |
| 53 | +- Your CRDs use unstructured/dynamic data without `x-kubernetes-preserve-unknown-fields` |
| 54 | +- You need upgrade tolerance when controller and CRD versions are temporarily mismatched |
| 55 | + |
| 56 | +## How Kubebuilder scaffold handles CRD ordering |
| 57 | + |
| 58 | +### Makefile Targets (`make install` + `make deploy`) |
| 59 | + |
| 60 | +```bash |
| 61 | +make install # Installs CRDs into the cluster |
| 62 | +make deploy # Deploys the controller |
| 63 | +``` |
| 64 | + |
| 65 | +Two separate commands. CRDs are installed first and established before the controller starts. Order is guaranteed. |
| 66 | + |
| 67 | +### YAML bundle distribution (make build-installer) |
| 68 | + |
| 69 | +```bash |
| 70 | +kubectl apply -f dist/install.yaml |
| 71 | +``` |
| 72 | + |
| 73 | +The bundle positions CRDs early in the file, after Namespace and before Deployment. This works for new installations. |
| 74 | + |
| 75 | +During upgrades, if the CRD already exists and the update fails or is slow, the Deployment may still update. The new controller may start before the CRD update completes. |
| 76 | + |
| 77 | +### Helm chart distribution |
| 78 | + |
| 79 | +By using the [Helm plugin](../plugins/available/helm-v2-alpha.md), you can distribute your solution as a Helm chart package. Users can install or upgrade it with: |
| 80 | + |
| 81 | +```bash |
| 82 | +helm install my-operator ./dist/chart |
| 83 | +``` |
| 84 | + |
| 85 | +Kubebuilder places CRDs in `templates/crd/` to ensure they upgrade with the controller. Helm has a built-in resource order that helps during installation. |
| 86 | + |
| 87 | +During upgrades, if the CRD update fails or is slow to propagate, Helm may still update the Deployment. |
| 88 | + |
| 89 | +Moreover, users can skip CRD updates with `helm upgrade --set crd.enable=false`. |
| 90 | + |
| 91 | +<aside class="note"> |
| 92 | + |
| 93 | +Helm recommends a `crds/` directory, but CRDs there **never upgrade**. Kubebuilder uses `templates/crd/` to keep CRDs in sync with controllers but the order is **best effort**. See [Why CRDs are added under templates](../plugins/available/helm-v2-alpha.md#why-crds-are-added-under-templates) for details. |
| 94 | + |
| 95 | +</aside> |
| 96 | + |
| 97 | +<aside class="note"> |
| 98 | +<h1>Upgrade and Lifecycle Considerations</h1> |
| 99 | + |
| 100 | +During upgrades, CRD updates may not complete before the controller starts. This can happen when deployment tools apply resources without explicit ordering mechanisms. |
| 101 | + |
| 102 | +CRD updates may hit errors or take time to propagate through the API server. If the controller Deployment updates at the same time, the new controller pod may start before the CRD update finishes. With strict validation enabled, controller writes will fail until the CRD is ready. |
| 103 | + |
| 104 | +Ordering tools like ArgoCD sync waves or FluxCD dependsOn can help by controlling when resources are applied. However, these control timing, not whether the CRD update succeeds. If not properly configured or if the CRD apply encounters issues, the controller may still deploy. |
| 105 | + |
| 106 | +For controllers that use CRDs from third-party operators (cert-manager, prometheus-operator, etc.), those CRDs have independent lifecycles. The third-party operator may upgrade its CRDs separately from your controller, which can lead to version mismatches. |
| 107 | + |
| 108 | +</aside> |
| 109 | + |
| 110 | +## Wiring an opt-in flag in cmd/main.go |
| 111 | + |
| 112 | +This feature is **not scaffolded by default**. Follow these steps to add it manually. |
| 113 | + |
| 114 | +### Step 1: Add the strictManager wrapper |
| 115 | + |
| 116 | +In `cmd/main.go`, add this type definition after the `init()` function: |
| 117 | + |
| 118 | +```go |
| 119 | +// strictManager wraps the manager to reject unknown fields instead of silently dropping them. |
| 120 | +// When the controller writes a field that doesn't exist in the CRD, the write fails immediately. |
| 121 | +// This helps catch typos and version mismatches between your code and cluster CRDs. |
| 122 | +type strictManager struct { |
| 123 | + ctrl.Manager |
| 124 | + strictClient client.Client |
| 125 | +} |
| 126 | + |
| 127 | +func (m *strictManager) GetClient() client.Client { |
| 128 | + return m.strictClient |
| 129 | +} |
| 130 | +``` |
| 131 | + |
| 132 | +### Step 2: Add required imports |
| 133 | + |
| 134 | +Add these imports to `cmd/main.go`: |
| 135 | + |
| 136 | +```go |
| 137 | +import ( |
| 138 | + // ... your existing imports ... |
| 139 | + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" |
| 140 | + "sigs.k8s.io/controller-runtime/pkg/client" |
| 141 | +) |
| 142 | +``` |
| 143 | + |
| 144 | +### Step 3: Add the command-line flag |
| 145 | + |
| 146 | +In the `main()` function, where other flags are defined, add: |
| 147 | + |
| 148 | +```go |
| 149 | +func main() { |
| 150 | + var metricsAddr string |
| 151 | + var enableLeaderElection bool |
| 152 | + var probeAddr string |
| 153 | + var strictFieldValidation bool // Add this |
| 154 | + |
| 155 | + flag.StringVar(&metricsAddr, "metrics-bind-address", ":8080", "...") |
| 156 | + flag.StringVar(&probeAddr, "health-probe-bind-address", ":8081", "...") |
| 157 | + flag.BoolVar(&enableLeaderElection, "leader-elect", false, "...") |
| 158 | + |
| 159 | + // Add this flag |
| 160 | + flag.BoolVar(&strictFieldValidation, "strict-field-validation", false, |
| 161 | + "Reject unknown fields instead of dropping them.") |
| 162 | + |
| 163 | + // ... rest of your code ... |
| 164 | +} |
| 165 | +``` |
| 166 | + |
| 167 | +### Step 4: Wrap the manager conditionally |
| 168 | + |
| 169 | +After creating the manager with `ctrl.NewManager()`, add this wrapper logic: |
| 170 | + |
| 171 | +```go |
| 172 | +mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{ |
| 173 | + Scheme: scheme, |
| 174 | + // ... your other options ... |
| 175 | +}) |
| 176 | +if err != nil { |
| 177 | + setupLog.Error(err, "unable to start manager") |
| 178 | + os.Exit(1) |
| 179 | +} |
| 180 | + |
| 181 | +// When enabled, the controller rejects writes with unknown fields instead of silently dropping them. |
| 182 | +var finalMgr ctrl.Manager = mgr |
| 183 | +if strictFieldValidation { |
| 184 | + finalMgr = &strictManager{ |
| 185 | + Manager: mgr, |
| 186 | + strictClient: client.WithFieldValidation( |
| 187 | + mgr.GetClient(), |
| 188 | + metav1.FieldValidationStrict, |
| 189 | + ), |
| 190 | + } |
| 191 | +} |
| 192 | + |
| 193 | +// Use finalMgr for all subsequent setup |
| 194 | +if err := (&controller.MyReconciler{ |
| 195 | + Client: finalMgr.GetClient(), |
| 196 | + Scheme: finalMgr.GetScheme(), |
| 197 | +}).SetupWithManager(finalMgr); err != nil { |
| 198 | + setupLog.Error(err, "unable to create controller", "controller", "My") |
| 199 | + os.Exit(1) |
| 200 | +} |
| 201 | + |
| 202 | +// Continue using finalMgr for health checks, starting manager, etc. |
| 203 | +if err := finalMgr.AddHealthzCheck("healthz", healthz.Ping); err != nil { |
| 204 | + setupLog.Error(err, "unable to set up health check") |
| 205 | + os.Exit(1) |
| 206 | +} |
| 207 | + |
| 208 | +if err := finalMgr.Start(ctrl.SetupSignalHandler()); err != nil { |
| 209 | + setupLog.Error(err, "problem running manager") |
| 210 | + os.Exit(1) |
| 211 | +} |
| 212 | +``` |
| 213 | + |
| 214 | +[client-field-validation-docs]: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/client#WithFieldValidation |
0 commit comments