Kernel observability and diagnosis

Baseline: v4.2.5. Pod/port health, successful SQL, commit readiness and synchronized data are separate signals. Preserve identifiers/evidence before diagnosing stages.

Evidence

Record client time, SQL/parameters, DDL, scale, build/config, CN/TN/service IDs, transaction/query IDs and error number/SQLSTATE. Correlate metrics, service logs and selected profiles in the same window. Redact credentials/data while preserving join identifiers.

Symptom

Separate first

Evidence/code domain

Slow query

Scans/skew, CPU, memory/spill, storage, backpressure

Plan/runtime statistics, Pipeline/object reads

Slow commit

Locks, preparation, WAL, quorum/network

Transaction/lock logs, metrics, wait profiles

Another CN misses a write

Source commit, snapshot, subscription/watermark, objects

Logtail metrics/logs and table/transaction IDs

Memory/disk pressure

Accounted/native allocations, spill disk/FD, retained history

Terminal accounts, spill, OS resources, GC

Slow/failed restart

Bootstrap, Checkpoint, Replay, Catalog, readiness

Stage logs, checkpoint/LSN, storage errors

Space stays allocated

Merge/GC and Snapshot/PITR/branch protection

Objects, background progress and protection state

Metrics and profiles

Metric names/labels follow pkg/util/metric/v2 and target-build dashboards. Transaction, Logtail, Pipeline and FileService families differ; RSS is not an allocation ledger. Correlate rates, counters and latency distributions in matching windows instead of one instantaneous value.

mo-service --debug-http exposes pprof; block/mutex sampling needs its parameters. Deployment controls sampling overhead, listen address and collection permissions. Select CPU/allocation/contention/wait profiles after identifying the domain; enabling profiling is not itself a repair.

Workflow and completion

  1. Minimize SQL/workload and establish results/failure.

  2. Identify the stage, change one condition and compare identical metrics.

  3. Inspect fixed-source wait/allocation/release points; distinguish correlation from proof.

  4. Verify results, latency/throughput and terminal resources, including cancellation/retry.

Invariant: diagnosis must preserve recovery evidence. Resources release according to their lifetimes; cache warming, task catch-up and history protection have distinct completion conditions.

See profiling parameters, EXPLAIN and resource monitoring.

Source and regression entry points