Context
An operator upgrade can change more than a Deployment image. It can introduce a new CustomResourceDefinition, alter accepted fields, change stored versions, and update the controller interpreting existing objects. Rolling back the chart does not automatically reverse every one of those changes. A schema and controller upgrade needs a compatibility plan of its own.
Consider an illustrative cluster running a database operator. A chart upgrade succeeds, but the new controller writes fields the previous version does not understand. The team rolls back the controller and discovers that the custom resources and managed database state have already moved forward. The recovery problem is larger than a Helm release revision.
Distinguish the definition from the resources
A CRD extends the Kubernetes API with a resource type; individual custom resources use that type. Helm documents the need to register a definition before creating its resources and describes the special handling and lifecycle caveats of CRDs shipped in the crds directory. Consult the exact Helm version and chart procedure rather than assuming CRDs follow ordinary template upgrade and rollback behaviour. 1
Inventory the CRD, controller, admission or conversion webhooks, custom resources, and managed external state. The upgrade sequence should name each component and its owner. A chart name alone does not describe the full system being changed.
Read the operator's supported upgrade path
Use the operator's release documentation for version compatibility, skipped-version restrictions, conversion requirements, and backup prerequisites. A chart may package a controller without making every historical jump supported. Pin chart and image versions so the reviewed path is the one actually deployed.
Identify whether the operator updates managed applications or databases automatically after its own upgrade. That secondary reconciliation can create irreversible or slow-to-reverse changes. Keep those actions visible in the maintenance plan instead of treating the controller upgrade as an isolated platform operation.
Compare schemas and live objects
Review added, removed, required, defaulted, and validation-constrained fields. Compare the schemas with representative existing custom resources, including older or unusual configurations. A newly required field or stricter validation can prevent later updates even if the object already exists.
Inspect served and stored API versions through the supported Kubernetes procedure. Conversion webhooks introduce another dependency: their service, certificates, availability, and compatibility need testing. Do not remove an old version simply because the new controller starts; verify that the stored-object migration and client compatibility requirements have been satisfied.
Preserve the right recovery material
Export the relevant configuration and establish an application-appropriate backup of managed data. Restrict access because custom resources and secrets can contain sensitive information. A YAML export of a database resource is not a backup of the database's contents.
Record how to restore the controller, definitions, resources, and managed state together. Decide whether recovery uses rollback, forward repair, or restoration into a clean environment. The chosen route should reflect what the vendor supports and what has been exercised, rather than a generic expectation that every upgrade can be undone in place.
Test with representative custom resources
Build a staging exercise using the same upgrade path and representative resource shapes. Include the oldest supported configuration, a resource using optional features, and a normal current configuration. Observe controller reconciliation and the managed service's behaviour after the schema change.
A chart render or dry-run can catch some issues but cannot model every API registration and conversion effect. Test against an appropriate API server and operator environment. Keep the distinction between template validation and a real schema transition visible in the release evidence.
Coordinate the sequence and availability
Follow the supported order for CRDs, webhooks, controllers, and dependent resources. Verify readiness at each meaningful boundary before moving on. A controller Pod being Ready does not prove it can reconcile every existing custom resource or reach the systems it manages.
For shared operators, identify all namespaces and teams affected by a cluster-scoped definition change. A chart installed for one team may manage an API used by several others. Communicate the maintenance scope and test representative workloads from the affected population, using the organization's normal coordination process.
Rehearse the failure decision
Choose clear stop conditions: conversion failures, rejected updates, unhealthy managed services, or unexpected destructive plans. Establish who decides whether to pause, repair forward, or restore. Keep the old artifacts available, but do not call their availability a complete rollback strategy.
An illustrative recovery drill might restore a managed test database into an isolated environment using the documented backup, then reconcile its custom resource with the supported controller version. That proves a stronger outcome than reinstalling an older chart and hoping the data state still fits.
Document the final compatibility boundary
Record installed CRD versions, controller version, chart version, migrated resources, backup reference, and the recovery exercise. Update application onboarding instructions so new custom resources use the supported schema. Keep deprecated fields visible until their planned removal rather than allowing silent drift between teams.
A focused operator exercise: A schema-compatibility acceptance sample
Prepare representative custom resources from the oldest supported configuration and the current configuration. Upgrade through the vendor-supported sequence in staging, then perform a harmless update to each resource and observe controller reconciliation. This catches cases where an object survives storage but later edits fail under the new validation rules.
If the recovery plan includes an older controller, exercise that path only as supported and verify the managed service and stored resource state remain compatible. Otherwise demonstrate the chosen forward-repair or isolated-restore route. Keep chart, controller, CRD, webhook, and managed-data versions together so the result describes the whole upgrade rather than one successful Helm command.
Quick diagnostic map
| Evidence | Review | Acceptance boundary |
|---|---|---|
| Schema version | Compare stored and served custom resources | Review compatibility before controller rollout |
| Upgrade ownership | Identify who manages CRD changes | Use an explicit migration sequence |
| Rollback boundary | Test older controller compatibility | Preserve recoverable custom-resource data |
Frequently asked questions
Will helm rollback automatically reverse every CRD and data change? Do not assume it. Verify the version-specific chart behaviour and the operator's supported recovery procedure.
Clouvenix can review the schema, controller, and managed data together, turning an operator upgrade into a sequence with measurable compatibility and recovery checks.
For a scoped review of this workflow, discuss the relevant Clouvenix engineering service.