Operations: checks, changes, and recovery¶
Manage MatrixOne as a verifiable service: inventory versions/topology, establish business baselines, drill restoration, and preserve evidence during changes and incidents. This page targets v4.2.5; single-node experiments are not production HA certification.
Maintain an operations inventory¶
Record MO tag/digest, Operator chart/controller image/CRDs, configuration source, UUIDs/Logservice hostname, CN/TN/Logservice/storage failure domains, client entry, identities, data/PVC/bucket/prefix, backup tools, and recovery procedures. Record credential references/owners, not plaintext secrets.
Frequency/event |
Check and deliverable |
|---|---|
Daily/each shift |
Connections/business writes, latency/errors, Pod restarts/resources, disk/inodes, storage errors, backup freshness |
Weekly |
Capacity trends, query regressions, Task/CDC backlog, privilege changes, certificate expiry, backup chains/retention |
Before every change |
Version/storage compatibility, recoverable backup, approved manifests, window, stop conditions, recovery plan |
Periodic recovery drill |
Independent-target restore, business validation, measured RPO/RTO, external dependencies, cutover plan |
Incident |
Freeze unrelated changes, identify affected layer, preserve timeline/errors/logs, contain and recover |
Alert on business impact¶
Category |
Signals |
First response |
|---|---|---|
Availability/errors |
Real-client connection/authentication failures, SQL error increase |
Separate endpoint/network/identity/database/storage causes |
Latency/congestion |
Key-query p95/p99, conflicts, concurrency/job backlog |
Compare traffic, plans, CPU/memory/IO, dependency changes |
Resource exhaustion |
OOMKilled, CPU throttling, disk/inode/spill exhaustion |
Preserve events and limits; reduce load rather than deleting state |
Recovery/data risk |
Stale successful backups, restore failure, missing chain, storage errors |
Validate completeness and independent restoration, not just job exit status |
Background work |
Task FAILED/TIMEOUT, CDC disconnection/backlog |
Inspect history, watermarks, and committed writes before idempotent retry |
Choose thresholds, duration, owner, and escalation from business SLOs/baselines. Use metric names/labels exposed by the running version; old screenshots/names are not validated rules. Low error rates do not establish data correctness; Pod Ready does not establish complete business readiness.
Incident diagnosis order¶
Reproduce from the real client and record time/timezone, identity/account, CN/endpoint, SQL, error code, retries, and impact.
Check recent version/configuration/schema/load/credential/network changes; distinguish one query, CN, account, and whole cluster.
Preserve status, restart/OOM/scheduling events, relevant process logs, and resource trends before changing settings or restarting.
Check routing/authentication for connection failures, plans/resources for latency, privileges/transactions/storage for writes, and version/format/identity/backup chains for recovery.
Apply an approved repair, verify business behavior, restore traffic, and record cause, actual recovery time, and follow-ups.
MO_NAMESPACE=mo
kubectl get pods,pvc,svc -n "$MO_NAMESPACE" -o wide
kubectl get events -n "$MO_NAMESPACE" --sort-by=.lastTimestamp
Select the affected Pod before collecting logs. Do not publicly bundle Secrets, credential-bearing configuration, or customer data. Access debug interfaces on authorized networks and retain the actual deployment/port details.
Operational entry points¶
This review verified single-CN version, writes, restart, fixed-hostname recreation, and SQL snapshot restore. It did not deploy a production Kubernetes cluster, drill multi-node failure, or perform cross-version upgrades. Production needs its own compatibility and recovery evidence.