Kernel observability and diagnosis¶
Baseline: v4.2.5. Pod/port health, successful SQL, commit readiness and synchronized data are separate signals. Preserve identifiers/evidence before diagnosing stages.
Evidence¶
Record client time, SQL/parameters, DDL, scale, build/config, CN/TN/service IDs, transaction/query IDs and error number/SQLSTATE. Correlate metrics, service logs and selected profiles in the same window. Redact credentials/data while preserving join identifiers.
Symptom |
Separate first |
Evidence/code domain |
|---|---|---|
Slow query |
Scans/skew, CPU, memory/spill, storage, backpressure |
Plan/runtime statistics, Pipeline/object reads |
Slow commit |
Locks, preparation, WAL, quorum/network |
Transaction/lock logs, metrics, wait profiles |
Another CN misses a write |
Source commit, snapshot, subscription/watermark, objects |
Logtail metrics/logs and table/transaction IDs |
Memory/disk pressure |
Accounted/native allocations, spill disk/FD, retained history |
Terminal accounts, spill, OS resources, GC |
Slow/failed restart |
Bootstrap, Checkpoint, Replay, Catalog, readiness |
Stage logs, checkpoint/LSN, storage errors |
Space stays allocated |
Merge/GC and Snapshot/PITR/branch protection |
Objects, background progress and protection state |
Metrics and profiles¶
Metric names/labels follow pkg/util/metric/v2 and target-build dashboards. Transaction, Logtail, Pipeline and FileService families differ; RSS is not an allocation ledger. Correlate rates, counters and latency distributions in matching windows instead of one instantaneous value.
mo-service --debug-http exposes pprof; block/mutex sampling needs its parameters. Deployment controls sampling overhead, listen address and collection permissions. Select CPU/allocation/contention/wait profiles after identifying the domain; enabling profiling is not itself a repair.
Workflow and completion¶
Minimize SQL/workload and establish results/failure.
Identify the stage, change one condition and compare identical metrics.
Inspect fixed-source wait/allocation/release points; distinguish correlation from proof.
Verify results, latency/throughput and terminal resources, including cancellation/retry.
Invariant: diagnosis must preserve recovery evidence. Resources release according to their lifetimes; cache warming, task catch-up and history protection have distinct completion conditions.
See profiling parameters, EXPLAIN and resource monitoring.