WAL and the commit pipeline

Baseline: v4.2.5. WAL carries transaction commands and context needed for recovery; LogService is its distributed backend. Logtail serves a different purpose: synchronizing readable TN state to CNs.

Commit pipeline

workspace → prepare/decide → generate and append WAL
                              │
                              ├→ collect Logtail
                              └→ wait for WAL and collection → ApplyCommit → finish

TxnManager.onWal calls PrepareWAL; onApply waits through WaitWalAndTail before applying the commit. Queues overlap transactions and I/O, but sending an append request is not durable completion. Bulk data objects also become visible through committed transaction metadata.

Independent progress markers

Marker

Meaning

Does not establish

WAL completion

Backend successfully processes the recovery record

All rows have been flushed into objects

Applied Logtail watermark

CN has synchronized state through a timestamp

A CDC destination or external client has committed

Checkpoint completion

Recovery information/object references form a recovery starting point

Internal files may be deleted despite retention protections

LSN identifies log position; transaction timestamps order visibility. They are not interchangeable. Logstore drivers maintain upper-level ordering, backend LSN mappings, truncation and replay boundaries.

Backend, truncation and recovery

The backend exposes append, read and truncation. LogService uses Raft and returns append results after quorum commit/application. Truncation requires a safe recovery basis, not merely a successful transaction or one CN’s consumed watermark.

Recovery loads checkpoint information and replays required WAL commands to reconstruct Catalog, object/MVCC state and progress. Preserve logs and objects on corruption, missing files or unknown transaction states; deleting internal directories is not a routine repair.

Diagnosis and verification

Separate lock waits, preparation, WAL waits, network latency and log quorum when diagnosing slow commits. Record transaction ID, timestamps, LSN, service ID, build and storage errors. Correlate WaitWalAndTail with transaction metrics, profiles and recovery logs.

A recovery experiment should restart an isolated instance after acknowledged commits and compare rows, including uncommitted work and repeated restarts. Successful SQL does not validate crash durability.

Source and regression entry points