Operations: checks, changes, and recovery

Manage MatrixOne as a verifiable service: inventory versions/topology, establish business baselines, drill restoration, and preserve evidence during changes and incidents. This page targets v4.2.5; single-node experiments are not production HA certification.

Maintain an operations inventory

Record MO tag/digest, Operator chart/controller image/CRDs, configuration source, UUIDs/Logservice hostname, CN/TN/Logservice/storage failure domains, client entry, identities, data/PVC/bucket/prefix, backup tools, and recovery procedures. Record credential references/owners, not plaintext secrets.

Frequency/event

Check and deliverable

Daily/each shift

Connections/business writes, latency/errors, Pod restarts/resources, disk/inodes, storage errors, backup freshness

Weekly

Capacity trends, query regressions, Task/CDC backlog, privilege changes, certificate expiry, backup chains/retention

Before every change

Version/storage compatibility, recoverable backup, approved manifests, window, stop conditions, recovery plan

Periodic recovery drill

Independent-target restore, business validation, measured RPO/RTO, external dependencies, cutover plan

Incident

Freeze unrelated changes, identify affected layer, preserve timeline/errors/logs, contain and recover

Alert on business impact

Category

Signals

First response

Availability/errors

Real-client connection/authentication failures, SQL error increase

Separate endpoint/network/identity/database/storage causes

Latency/congestion

Key-query p95/p99, conflicts, concurrency/job backlog

Compare traffic, plans, CPU/memory/IO, dependency changes

Resource exhaustion

OOMKilled, CPU throttling, disk/inode/spill exhaustion

Preserve events and limits; reduce load rather than deleting state

Recovery/data risk

Stale successful backups, restore failure, missing chain, storage errors

Validate completeness and independent restoration, not just job exit status

Background work

Task FAILED/TIMEOUT, CDC disconnection/backlog

Inspect history, watermarks, and committed writes before idempotent retry

Choose thresholds, duration, owner, and escalation from business SLOs/baselines. Use metric names/labels exposed by the running version; old screenshots/names are not validated rules. Low error rates do not establish data correctness; Pod Ready does not establish complete business readiness.

Incident diagnosis order

  1. Reproduce from the real client and record time/timezone, identity/account, CN/endpoint, SQL, error code, retries, and impact.

  2. Check recent version/configuration/schema/load/credential/network changes; distinguish one query, CN, account, and whole cluster.

  3. Preserve status, restart/OOM/scheduling events, relevant process logs, and resource trends before changing settings or restarting.

  4. Check routing/authentication for connection failures, plans/resources for latency, privileges/transactions/storage for writes, and version/format/identity/backup chains for recovery.

  5. Apply an approved repair, verify business behavior, restore traffic, and record cause, actual recovery time, and follow-ups.

MO_NAMESPACE=mo
kubectl get pods,pvc,svc -n "$MO_NAMESPACE" -o wide
kubectl get events -n "$MO_NAMESPACE" --sort-by=.lastTimestamp

Select the affected Pod before collecting logs. Do not publicly bundle Secrets, credential-bearing configuration, or customer data. Access debug interfaces on authorized networks and retain the actual deployment/port details.

Operational entry points

This review verified single-CN version, writes, restart, fixed-hostname recreation, and SQL snapshot restore. It did not deploy a production Kubernetes cluster, drill multi-node failure, or perform cross-version upgrades. Production needs its own compatibility and recovery evidence.