Distributed execution, memory and spill¶
Baseline: v4.2.5. Compile turns logical plans into Scopes, Pipelines and vectorized operators. Actual placement determines other-CN execution; CN count/cache hits do not prove it.
Runtime graph¶
scan → filter/project → local operators → exchange/shuffle → join/aggregate → merge
│ │
└→ Batch └→ CN pipeline messages
Scope organizes fragments, Pipeline operator chains and Batch/Vector column data. Broadcast and Shuffle differ in copying/partitioning cost; skew can bottleneck workers. The logical plan is not the complete physical graph.
Waiting and cancellation¶
Producers/consumers, MessageBoard, remote RPC and result return can wait on each other. Completion, errors, cancellation and unreachable nodes must terminate waits and release batches, JoinMaps, connections and messages. A result termination marker alone may not prove all server resources are released.
Invariant: one attempt’s messages/resources cannot enter a later prepared execution or retry; cancellation needs a termination path. Separate TCP/FD pressure, backpressure and skew.
Allocation ownership and scope¶
HashBuild/join consumers have physical allocation accounting. An allocation records account/owner/site and releases its charge on free. Views borrow without charging twice; copies allocate to the destination owner. Attempts drain messages/operators before sealing/finalizing accounts.
This controlled domain is not total query RSS. Go/native libraries without allocator hooks, some native indexes and GPU storage have separate boundaries. Remote fragments of one statement also share boards and aggregate terminal state.
Pressure and spill¶
Memory, spilled disk bytes and open FDs are separate resources. Optional Runtime Filters may degrade to PASS; required join state cannot silently bypass admission. Capacity pressure differs from sealed-account/ownership errors, which are not retryable memory shortages.
Spill moves data/resource ownership; encoding, partitioning, decoding and rebuilding still need controlled memory. Retrying requires progress such as reclaiming, advancing partitions, shrinking input or degrading optional state, not repeating identical I/O indefinitely.
Observation¶
Record physical plans, actual remote fragments, distributions, operator memory/spill, temporary disk, FDs, cancellation and terminal resources. sort_spill_mem controls a sorting path, not all HashBuild memory.