TAE storage layout and MVCC¶
Baseline: v4.2.5. TAE on TN manages transactions, Catalog/MVCC, objects, WAL and background work. DistTAE on CN manages the workspace, synchronized state and distributed reads.
Data units¶
Unit |
Role |
Entry point |
|---|---|---|
Vector / Batch |
Column vectors and operator batches |
|
Block |
Columnar data within objects and row addressing |
|
Object |
Immutable persisted data and metadata |
|
Tombstone |
Deletions and their persisted representation |
TAE object state and CN deletion merging |
Catalog / MVCC |
Transaction lifetimes of databases, tables, schemas and objects |
|
Historical three-level LSM/Segment diagrams are background, not a description of the complete current Object/Tombstone lifecycle. Column Family DDL switching between row/column storage is not advertised as an available capability. Check implementation/configuration for object sizes, block rows and Merge policies.
Writes and deletes¶
CN workspace → TN preparation → WAL → commit
│ │
└→ bulk object └→ memory/object state → Logtail
│
└→ Flush / Merge / Checkpoint
Small committed writes may remain in memory; CN may produce objects for bulk writes. Uploading an object is not committing its transaction. UPDATE can become delete plus insert; follow row IDs, primary-key deduplication and tombstones rather than assuming in-place object updates.
Snapshot reads¶
Reads select timestamp-visible objects, read column blocks and combine memory rows, tombstones and workspace changes. MVCC and retention govern whether an old snapshot may still read objects replaced by Merge; a new query’s object list is not a deletion criterion.
Invariant: layout changes preserve the logical result for the corresponding SQL snapshot. Object indexes/ZoneMaps help filtering but do not independently implement all visibility and primary-key constraints.
Flush, Merge and resources¶
Flush persists appropriate memory data/deletions. Merge reorganizes objects, sorting/deletions and metadata replacement. Checkpoint records the recovery basis. Different queues, transactions and I/O participate; a COMMIT does not predict their completion.
High deletion ratios, small objects, overlapping ranges, long transactions and protected history can affect scans and reclamation. Inspect objects/blocks scanned, bytes read, deletion state and background work instead of deriving storage usage from row count alone.
Failures and diagnosis¶
For slow commits separate locking/deduplication, preparation and WAL. For slow reads separate pruning, deletion merging, caches and object I/O. For retained storage distinguish pending Merge, GC progress and history protection. For slow restart separate checkpoint loading and WAL replay. Retain transaction/table/object IDs and time ranges.
See WAL, Logtail and transactions/locks. The 2022 RFC is historical background, not the current feature inventory.