WAL and the commit pipeline¶
Baseline: v4.2.5. WAL carries transaction commands and context needed for recovery; LogService is its distributed backend. Logtail serves a different purpose: synchronizing readable TN state to CNs.
Commit pipeline¶
workspace → prepare/decide → generate and append WAL
│
├→ collect Logtail
└→ wait for WAL and collection → ApplyCommit → finish
TxnManager.onWal calls PrepareWAL; onApply waits through WaitWalAndTail before applying the commit. Queues overlap transactions and I/O, but sending an append request is not durable completion. Bulk data objects also become visible through committed transaction metadata.
Independent progress markers¶
Marker |
Meaning |
Does not establish |
|---|---|---|
WAL completion |
Backend successfully processes the recovery record |
All rows have been flushed into objects |
Applied Logtail watermark |
CN has synchronized state through a timestamp |
A CDC destination or external client has committed |
Checkpoint completion |
Recovery information/object references form a recovery starting point |
Internal files may be deleted despite retention protections |
LSN identifies log position; transaction timestamps order visibility. They are not interchangeable. Logstore drivers maintain upper-level ordering, backend LSN mappings, truncation and replay boundaries.
Backend, truncation and recovery¶
The backend exposes append, read and truncation. LogService uses Raft and returns append results after quorum commit/application. Truncation requires a safe recovery basis, not merely a successful transaction or one CN’s consumed watermark.
Recovery loads checkpoint information and replays required WAL commands to reconstruct Catalog, object/MVCC state and progress. Preserve logs and objects on corruption, missing files or unknown transaction states; deleting internal directories is not a routine repair.
Diagnosis and verification¶
Separate lock waits, preparation, WAL waits, network latency and log quorum when diagnosing slow commits. Record transaction ID, timestamps, LSN, service ID, build and storage errors. Correlate WaitWalAndTail with transaction metrics, profiles and recovery logs.
A recovery experiment should restart an isolated instance after acknowledged commits and compare rows, including uncommitted work and repeated restarts. Successful SQL does not validate crash durability.