Distributed execution, memory and spill

Baseline: v4.2.5. Compile turns logical plans into Scopes, Pipelines and vectorized operators. Actual placement determines other-CN execution; CN count/cache hits do not prove it.

Runtime graph

scan → filter/project → local operators → exchange/shuffle → join/aggregate → merge
                              │                  │
                              └→ Batch           └→ CN pipeline messages

Scope organizes fragments, Pipeline operator chains and Batch/Vector column data. Broadcast and Shuffle differ in copying/partitioning cost; skew can bottleneck workers. The logical plan is not the complete physical graph.

Waiting and cancellation

Producers/consumers, MessageBoard, remote RPC and result return can wait on each other. Completion, errors, cancellation and unreachable nodes must terminate waits and release batches, JoinMaps, connections and messages. A result termination marker alone may not prove all server resources are released.

Invariant: one attempt’s messages/resources cannot enter a later prepared execution or retry; cancellation needs a termination path. Separate TCP/FD pressure, backpressure and skew.

Allocation ownership and scope

HashBuild/join consumers have physical allocation accounting. An allocation records account/owner/site and releases its charge on free. Views borrow without charging twice; copies allocate to the destination owner. Attempts drain messages/operators before sealing/finalizing accounts.

This controlled domain is not total query RSS. Go/native libraries without allocator hooks, some native indexes and GPU storage have separate boundaries. Remote fragments of one statement also share boards and aggregate terminal state.

Pressure and spill

Memory, spilled disk bytes and open FDs are separate resources. Optional Runtime Filters may degrade to PASS; required join state cannot silently bypass admission. Capacity pressure differs from sealed-account/ownership errors, which are not retryable memory shortages.

Spill moves data/resource ownership; encoding, partitioning, decoding and rebuilding still need controlled memory. Retrying requires progress such as reclaiming, advancing partitions, shrinking input or degrading optional state, not repeating identical I/O indefinitely.

Observation

Record physical plans, actual remote fragments, distributions, operator memory/spill, temporary disk, FDs, cancellation and terminal resources. sort_spill_mem controls a sorting path, not all HashBuild memory.

Source and regression entry points