Kubernetes Operator acceptance operations¶
This guide complements Operator management and production readiness. It references chart, CRDs and controllers at Operator commit 10aef15c57f98f92b442d79810f8ad3d53bbd85b; examples do not certify every MO/Kubernetes combination.
Versions and offline preparation¶
Record Kubernetes server version, MO image digest, Operator chart/version, controller image, backup image and CRD source separately. Chart appVersion is not the database version. Confirm StorageClass, object-storage endpoint/separate prefix, same-namespace Secret, registry access and webhook reachability.
Offline installation needs the chart/dependencies, all controller/backup/Kruise/MO images and credentials configuration, not only the MO image. Prepare operator-values.yaml for the selected chart (images, resources, scheduling and watch scope), then render:
export OP_NS=matrixone-operator
export MO_NS=mo
export OP_RELEASE=mo-operator
export MO_CLUSTER=mo
export OP_CHART=/path/to/approved/matrixone-operator.tgz
helm show chart "$OP_CHART"
helm show values "$OP_CHART" > operator-defaults.yaml
helm template "$OP_RELEASE" "$OP_CHART" -n "$OP_NS" \
-f operator-values.yaml > operator-rendered.yaml
Inspect images, RBAC, Services/Webhooks, CRDs and watch scope. With onlyWatchReleasedNS, ensure the MO namespace is watched. Rendering does not validate cluster networking or reconciliation. Install with:
helm upgrade --install "$OP_RELEASE" "$OP_CHART" -n "$OP_NS" --create-namespace -f operator-values.yaml --wait --timeout 10m
Before upgrading, save helm get values --all, helm get manifest, CRs and CRDs. Check how the chosen chart delivers CRDs (templates or crds directory); do not assume helm upgrade migrates every CRD. Review schema/default changes, resource recreation and MO restart effects. Helm rollback does not roll back database data or CRD schemas.
Apply CRs and verify readiness¶
Start from the pinned cluster example. Replace MO version, storage, replicas, resources and scheduling. Example defaults are not a production capacity plan. Use server dry-run on the target cluster first:
kubectl -n "$MO_NS" apply --dry-run=server -f mo.yaml
kubectl -n "$MO_NS" apply -f mo.yaml
helm list -n "$OP_NS"
kubectl -n "$OP_NS" get deploy,pods
kubectl -n "$MO_NS" get matrixonecluster "$MO_CLUSTER" -o yaml
kubectl -n "$MO_NS" get cnsets,dnsets,logsets,pods,svc,pvc
kubectl -n "$MO_NS" get events --sort-by=.metadata.creationTimestamp
Check controller availability, CR conditions/phase, desired/ready component replicas, Pod events, PVCs and storage access separately. Helm deployed, Pod Running or Ready conditions do not replace client/SQL acceptance. Connect through the actual Service/Proxy, create and populate a dedicated test table, then read back through an independent connection. Record version and expected output.
Troubleshooting¶
Symptom |
First checks |
|---|---|
Rejected CR |
CRD apiVersion/schema, webhook Service/endpoints/certificate and validation errors |
CR never ready |
conditions reason/message, observedGeneration if available, controller logs and watch scope |
Pending Pod |
Scheduling events, requests, taints/affinity, PVC/StorageClass |
ImagePullBackOff |
Image tag/digest, node registry access and namespace of imagePullSecrets |
CrashLoopBackOff |
|
Connection failure |
Service/endpoints, Proxy/CN readiness, NetworkPolicy, credentials and listen port |
Get the actual controller deployment name using kubectl get deploy, rather than guessing from the first Pod. Edit owning CRs/Helm values instead of managed Pods, which reconciliation can overwrite. Check capacity and connection behavior before scaling CN; do not present TN/LogService downscaling as a generic operation.
Backup, restore and success criteria¶
Replace namespace, source cluster, storage endpoint/prefix and Secret in this structural example:
apiVersion: core.matrixorigin.io/v1alpha1
kind: BackupJob
metadata:
name: acceptance-backup
namespace: mo
spec:
source:
clusterRef: mo
target:
s3:
type: minio
endpoint: http://minio.default:9000
path: acceptance/backup-01
secretRef:
name: minio
kubectl -n "$MO_NS" apply --dry-run=server -f backup.yaml
kubectl -n "$MO_NS" apply -f backup.yaml
kubectl -n "$MO_NS" wait --for=condition=Ended backupjob/acceptance-backup --timeout=10m
kubectl -n "$MO_NS" get backupjob acceptance-backup -o yaml
kubectl -n "$MO_NS" get backups
Ended=True can indicate failure. The current controller sets phase=Completed and status.backup on success, or phase=Failed on failure. Inspect the Backup reference/errors, then restore into a separate cluster and storage prefix with a supported MO version. restoreFrom references a Backup resource, not a BackupJob name or mo_br UUID. Compare schema, keys, every field, NULLs and business queries. Record elapsed time and recovery point; successful startup does not prove correct data.
Deletion and retention¶
Deleting MatrixOneCluster triggers cleanup of managed resources. Record ownerReferences/finalizers, PVC deletion policy, PV reclaimPolicy, Kruise retention settings, object-storage prefix and independent backup first. Namespace deletion also affects Secrets/Backup resources; uninstalling the Operator can stop reconciliation; deleting a CRD deletes its custom resources. Neither universal preservation nor universal deletion is guaranteed. Rehearse the exact version in an isolated cluster. Diagnose a blocked finalizer through the controller rather than force-removing it as a normal cleanup step.
Verification scope¶
YAML parsing, pinned CRD schema checks, Helm rendering and shell syntax checks are static checks. Server dry-run, deployment, failover, cross-version upgrades, restore and deletion require an authorized isolated Kubernetes cluster and separate evidence. Local static success is not production lifecycle certification.