Recovery selection, drills, and acceptance

A successful backup does not establish recoverability. Define the failure, acceptable data loss (RPO), and recovery time (RTO), select a method, and drill on an independent target.

Choose by failure

Scenario

Candidate

Boundary

Mistaken update/delete with intact storage

SQL Snapshot/PITR

Depends on retained objects/history in the same system; not an independent DR copy

Lost/unavailable object storage or site

Verified backup on independent media

Must not depend on the failed original bucket/prefix; target/tool/format compatibility required

Migration/selective restoration

Logical export/import

Verify objects, privileges, types, encoding, sequences, external dependencies; file exports are not full-instance backups

Physical full/incremental restoration

Compatible tool and complete chain

Retain base, ordered increments, checksums/keys, and tool version; follow actual CLI

SQL Snapshot/PITR, object-store version history, application backups, and mo_br physical backups differ. Seeing the database in SHOW DATABASES verifies its name, not restored data, privileges, or business behavior.

Independent-target drill

  1. Record failure time/timezone, source version/configuration, backup point/chain, checksums, key references, and scope.

  2. Prepare an independent namespace/volume/bucket prefix. Stop concurrent target writes, check tool version and --help, and follow its reference.

  3. Restore on the target and preserve errors; clearing production directories or shared buckets is not a diagnosis step.

  4. Check definitions, counts, key ranges, critical aggregates/watermarks, samples, application privileges, and external references. For CDC/Tasks, assess replay, duplicates, and recovery point.

  5. Validate business behavior from actual clients before switching. Measure lost/duplicate data and duration; planned backup frequency is not measured RPO.

Isolated SQL snapshot experiment

Run only on a test instance with unused names. This experiment restores its own temporary database to an earlier state. It verifies recovery from a mistaken write, not object-storage loss or physical-backup DR.

CREATE DATABASE mo_recovery_lab;
USE mo_recovery_lab;
CREATE TABLE ledger(id INT PRIMARY KEY, amount INT);
INSERT INTO ledger VALUES(1,42);
CREATE SNAPSHOT mo_recovery_lab_snap FOR DATABASE mo_recovery_lab;
UPDATE ledger SET amount=99 WHERE id=1;
-- Expected-Value: 99
SELECT amount FROM ledger WHERE id=1;
RESTORE DATABASE mo_recovery_lab {SNAPSHOT='mo_recovery_lab_snap'};
-- Expected-Value: 42
SELECT amount FROM mo_recovery_lab.ledger WHERE id=1;
DROP SNAPSHOT mo_recovery_lab_snap;
DROP DATABASE mo_recovery_lab;

The v4.2.5 probe restored using {SNAPSHOT='...'}. RESTORE DATABASE ... FROM SNAPSHOT ... returns 1064 in that version; do not copy another version’s compatibility syntax blindly. Check creation with SHOW SNAPSHOTS before modifying data.

Physical backups and retention

Use the mo_br reference and tool release guidance for availability, versions, commands, and compatibility. Historical tool names/logs do not establish current availability. This review did not obtain and execute the complete physical full/incremental toolchain and does not claim physical-backup verification.

Clean only expired backups managed by the tool after checking dependencies from retained bases/increments/snapshots. Do not manually delete shared objects, Raft/WAL, checkpoint files, or snapshot-referenced files. Isolate recovery targets from production and ensure backups do not disappear with primary storage.

Acceptance record

Save the recovery point/checks, target version, steps/errors, business validation, measured loss interval/duration, cutover/backout plan, owner, and follow-ups. Explicitly record unverified areas such as multi-CN restoration, privileges, external references, and cross-version recovery.