Upgrades: compatibility, changes, and recovery

An upgrade is more than changing an image tag. Verify database, Operator, CRD, backup-tool, and storage-format compatibility separately. The baseline is v4.2.5; this page does not certify direct/rolling upgrades from every historical release.

Before a change

Item

Prepare

Source/target

MO tags/digests, configuration, Operator chart/image/CRDs, tool versions

Compatibility

Release notes, supported path, schema/catalog and FileService format changes

Data/dependencies

PVCs, bucket/prefix, node identity, privileges, files, CDC/Tasks/client connections

Protection/validation

Recoverable independent backup, test-target restore, key counts/aggregates/read-write baseline

Change control

Window, traffic plan, owner, stop conditions, recovery target, expected interruption

Changing .spec.version may cause Operator-managed service replacement; it does not establish compatibility or uninterrupted availability. A new running process is not sufficient upgrade acceptance. Verify Operator/CRD changes separately from database version changes.

Execute in phases

  1. Pin source/target manifests/images; validate upgrade, recovery, and application compatibility in an independent test topology.

  2. Drill restoration and save data checks. Arrange write suspension or a maintenance window where required.

  3. Follow the supported version path and its Operator/database instructions and component order. This page does not invent a universal CN/TN/Logservice rolling order.

  4. Observe resource conditions, events, restarts, SQL errors, key queries, storage, and jobs; pause on stop conditions.

  5. Validate reads/writes, transactions, privileges, indexes/queries, external dependencies, and CDC/Task state from real clients before restoring traffic.

Start with these read-only checks, substituting actual resource names:

MO_NAMESPACE=mo
MO_CLUSTER=mo
kubectl get matrixonecluster "$MO_CLUSTER" -n "$MO_NAMESPACE" -o yaml
kubectl get pods,pvc,svc -n "$MO_NAMESPACE" -o wide
kubectl get events -n "$MO_NAMESPACE" --sort-by=.lastTimestamp

Review full YAML before sharing it; it can contain sensitive references. Apply approved manifests through the approved process rather than copying old nightly/chart or incompatible backend examples.

Configuration rollback differs from data recovery

Situation

Response

New image cannot be pulled and never ran

Repair image/scheduling configuration; verify old instances and data

New version already migrated catalog/schema/format

Changing the tag back is not a downgrade guarantee; use a supported path or restore

Business data error or object damage

Select a recovery point, restore independently, verify, then switch

Operator/CRD change

Check resource schema/controller compatibility separately; Helm rollback is not data recovery

Snapshot rollback of business data does not downgrade binaries, catalog, or file formats. Record allowed data loss, measured recovery time, and target version in the recovery plan.

Rebuild boundaries

Routine upgrade steps do not delete PVCs, original directories, or shared buckets. Redeploy to an independent namespace/volume/bucket prefix and retain the old environment for diagnosis/recovery. Only a separate retirement process after successful independent restoration, business acceptance, and an explicit decision to discard old data should remove old storage; it is not a routine upgrade prerequisite.

See backup methods and recovery drills. Cross-version and Kubernetes rolling upgrades were not executed in this review; no zero-interruption or arbitrary downgrade claim is made.