Design patterns and optimization · Intermediate

Vectorized execution

Process bounded batches to amortize execution overhead while distinguishing vectorized query execution from hardware SIMD.

Ask Logeia about this articleTry the three questionsReview your answers

Updated

On this page
  1. Amortize per-row execution overhead
  2. Trace a filter through a selection vector
  3. Distinguish batching, SIMD and parallel workers
  4. Bound batches to preserve locality and responsiveness
  5. Ask Logeia about this articleCheck your understanding

Amortize per-row execution overhead

Vectorized execution passes bounded batches of values between operators instead of invoking the entire operator pipeline separately for every row. During a million-row scan, repeated function calls, interpretation and type checks can consume substantial work beyond the arithmetic itself. A batch containing 1,024 values can amortize some control overhead across those values. The execution vector is a software batch; it does not mean that one CPU instruction processes all 1,024 rows simultaneously.

Trace a filter through a selection vector

For values [3, 8, 1, 9], filtering for values greater than five produces selected positions [1, 3]. A downstream operator can consult the original array through those positions or compact the survivors into [8, 9]. Selection vectors can avoid copying, but introduce indirect access. If nullness is stored in a separate validity bitmap, values, positions and validity must remain aligned. Rearranging values without the corresponding metadata can change query results rather than merely changing performance.

Distinguish batching, SIMD and parallel workers

Homogeneous contiguous values may make hardware SIMD easier to exploit, but variable-length strings, branch-heavy functions and dependent updates can limit that opportunity. Floating-point reductions can also produce different rounding when evaluation order changes. Software vectorization, SIMD instructions and multithreaded parallelism are separate design axes. A single-threaded batch engine may benefit from fewer calls without using SIMD, while a SIMD kernel may still operate inside a larger vectorized operator. Verify each claim independently.

Bound batches to preserve locality and responsiveness

Larger batches amortize more call overhead, but can enlarge intermediate state beyond cache capacity and delay the first result. A size such as 1,024 is an illustrative design choice, not a universal optimum. Column width, concurrency and operator state alter the trade-off. Compare identical query results while measuring peak intermediate memory and time to first output as well as total completion time. Tiny inputs can favor a simpler row-oriented path because setup costs dominate.

Batch boundaries are also opportunities to check cancellation. If a consumer stops after enough rows, continuing every remaining batch can waste substantial work. Decide how operators signal end-of-demand and how quickly expensive kernels can observe it. This is another reason to evaluate responsiveness along with throughput: minimizing per-row control overhead should not silently remove every practical opportunity to stop work that is no longer needed.

Reading and practice are recorded separately in this browser, for each language.

Learn this in order

Build your CS foundations · STEP 8

Prepare for AI systems · STEP 5

Choose a learning path