mo-tool advanced diagnostics and offline inspection¶
mo-tool is a development/diagnostic utility in the MatrixOne source tree, distinct from standalone mo_ctl. Commands below describe main; release packages may omit the binary or commands. Record tool and server commits and prefer matching builds.
Build and discover commands¶
Prepare the same Go, C/C++ and native dependencies as MO. From the source root:
make mo-tool
./mo-tool --help
./mo-tool inspect --help
./mo-tool debug goroutine --help
./mo-tool dashboard --help
./mo-tool object --help
./mo-tool ckp --help
inspect is an advanced online-maintenance entry. Read subcommand help, connection requirements and side effects first. Help output does not prove cluster execution.
Goroutines: capture, group, interpret¶
Capture text stacks from the configured debug endpoint:
curl -fsS 'http://127.0.0.1:12345/debug/pprof/goroutine?debug=2' -o goroutines.txt
./mo-tool debug goroutine
At the interactive prompt, enter:
parse goroutines.txt
count
group
top 10
exit
Repeat captures over time, compare growth in stack groups and correlate SQL, transactions, CPU and memory. One capture does not prove a leak; parse/count success does not establish root cause. See profiling flags for CPU/block/mutex collection.
Grafana dashboards¶
./mo-tool dashboard create --help
./mo-tool dashboard list --help
Configure the actual Grafana host, credentials, mode and datasource. After create, list UIDs and check real metric results in Grafana. Modes local/cloud/k8s/cloud-ctrl differ. Dashboard creation does not establish Prometheus collection. Deletion changes Grafana content; verify the UID/folder first.
Checkpoints and objects¶
Inspect a backup copy or protected inspection directory while preserving original files. Evaluate GC/checkpoint changes when reading active storage.
./mo-tool ckp info ./mo-data/shared2 --local2
./mo-tool ckp view ./mo-data/shared2 --local2
./mo-tool ckp show-create-table --table-id=272387 ./mo-data/shared2 --local2
./mo-tool ckp dump --table-id=272387 -o table.csv ./mo-data/shared2 --local2
The local example uses the DISK-V2 shared2 directory and –local2. Match your fileservice path/backend; CRC DISK does not use –local2. Checkpoint commands write logs in .mo-tool under the user directory and require a writable log location. The table ID is illustrative; find the actual ID in the viewer. Default CSV has no header and represents NULL as \N. Logical export applies tombstones at the selected checkpoint, excludes hidden columns and depends on catalog metadata. Fail rather than guess if metadata cannot be resolved. Historical timestamps use the version’s --ts PHYSICAL:LOGICAL. Lexical ordering buffers rows and suits smaller comparisons.
./mo-tool ckp info --fs-config etc/launch-minio-local/tn.toml --fs-name SHARED
./mo-tool object info object-name --fs-config etc/launch-minio-local/tn.toml --fs-name SHARED
Replace the config and object name. A remote key-prefix must point to the MO data root, not its ckp subdirectory: otherwise checkpoints may resolve while table objects cannot. See the source README for flags and internal-format boundaries.
Offline export is not a complete database backup: it does not promise all WAL, permissions, accounts or runtime configuration. Import into an independent target using the recovered schema; compare keys, fields, NULLs and the selected snapshot. Preserve export timestamp and tool version. Use backup/restore procedures for disaster recovery.