DBRaven
Pattern · data storage

Columnar Storage

mature

Summary

Physically store table data grouped by column rather than by row, so an analytical query that touches a fraction of a table's columns reads only those columns from disk, at the cost of making single-row reads and writes far more expensive than in a row-store.

Problem

A row-store forces an analytical query to read every column of every matching row even when the query only needs a handful of them, wasting I/O bandwidth and cache space on unread columns and capping scan throughput on wide tables. Analytical workloads that aggregate over billions of rows need a physical layout where the cost of a scan scales with the columns actually read, not the width of the table.

Description

A row-store (PostgreSQL heap, InnoDB clustered index) physically lays out all of a row's columns contiguously, because the common OLTP access pattern is "fetch or modify one row." A columnar engine (ClickHouse, Snowflake) inverts that layout: all values for one column across every row are stored contiguously, in their own file or block, because the common OLAP access pattern is "aggregate one or a few columns across many rows." A query that touches 3 of a table's 50 columns reads only those 3 columns' data from disk; a row-store reading the same query would read every column of every matching row and discard the other 47, wasting the I/O and cache space on data the query never needed.

Storing values column-wise also unlocks compression that row-wise storage cannot reach. Values within one column tend to be far more similar or repetitive than values across a row: a status column might have three distinct values across a billion rows, a timestamp column is monotonic or near-monotonic. Run-length encoding (store a value once plus a repeat count) and dictionary encoding (store each distinct value once and reference it by a small integer) both work far better against that column-wise regularity than they do against a row's mix of unrelated types and cardinalities. ClickHouse and Snowflake apply this compression per column, then execute scans in a vectorized fashion: operating on a batch of column values at once rather than one row at a time, which is both cache-friendly and amenable to SIMD instructions.

The same physical layout that makes wide scans cheap makes single-row operations expensive. Reading "one row" means touching a separate file or block for every column in that row and reassembling them, the inverse of the row-store's single contiguous read. Writing or updating "one row" is worse: it means touching every column's physical structure for a single logical change. ClickHouse's MergeTree engine avoids this entirely for the write path by batching inserts into immutable parts that are merged together in the background; a workload that sends many single-row inserts causes excessive part fragmentation and degraded read performance, because merge throughput cannot keep up with part creation. In-place row updates are not a native operation in ClickHouse at all: ALTER TABLE ... UPDATE and DELETE are asynchronous mutations that rewrite whole parts rather than modifying rows in place. Snowflake takes a related approach at the storage layer: data is organized into immutable micro-partitions with per-column metadata (min/max zone maps) that let a query prune micro-partitions that cannot match a predicate without reading them, but changing a single row still means writing a new micro-partition. Both designs are why columnar engines are built for OLAP and are a poor fit for OLTP point-query or point-update workloads.

Tradeoffs

Analytical scan throughput
+0.9

A query reads only the columns it touches, and vectorized, SIMD-friendly execution processes them in batches; scans over wide tables with selective column access are the pattern's primary strength.

Compression ratio
+0.8

Column-wise value regularity (low distinct-value counts, monotonic sequences) makes run-length and dictionary encoding far more effective than the same techniques applied row-wise; ClickHouse documents 5-10x compression versus raw size with LZ4/ZSTD.

Point-lookup / point-write latency
-0.7

Reading or writing a single row means touching a separate column file or block per column and reassembling or updating them individually, which is materially more expensive than a row-store's single contiguous access.

OLTP update-heavy workload fit
-0.7

In-place row updates are not native; ClickHouse mutations rewrite whole parts asynchronously and Snowflake writes a new micro-partition for any row change, making frequent per-row updates an expensive, poor-fit workload.

Batch ingest throughput
+0.6

Bulk-loaded or buffered batch inserts amortize the part/micro-partition creation and background merge cost across many rows, giving strong throughput when writes are batched rather than sent one row at a time.

When to use

OLAP or aggregation-heavy workload (GROUP BY, SUM, COUNT over large row counts)

Columnar layout reads only the columns an aggregation touches, and vectorized execution processes them in batches

Wide tables where individual queries access a small, selective subset of columns

The narrower the column subset relative to table width, the larger the I/O savings versus a row-store scan

Write pattern is batch-oriented (bulk loads, buffered ingestion) rather than per-row

Columnar engines are built around merging batches of immutable data; single-row inserts fight that design

When not to use

OLTP workload with frequent single-row point queries or point updates

Reading or writing one row touches many separate column structures, the inverse of a row-store's single contiguous access

High-frequency single-row writes that cannot be batched

Unbatched single-row inserts cause excessive part or micro-partition fragmentation and degrade both write and read performance

Operational Requirements

mandatory

Batch inserts; avoid sending single-row writes directly to the engine

Unbatched single-row inserts cause excessive part fragmentation (ClickHouse) or micro-partition churn (Snowflake) and degrade read performance

mandatory

Monitor part or micro-partition count and background merge/compaction backlog

A growing part count is the leading indicator that write rate is outpacing background merge throughput

recommended

Choose a sort key or clustering key that matches common query predicates

Zone-map and sparse-index pruning only skip data the sort order makes contiguous; a mismatched sort key forces full scans

recommended

Route point-lookup and point-update traffic to a row-store rather than the columnar engine

Mixed OLTP/OLAP workloads on one columnar engine push it outside its designed access pattern

Characteristics

Scales on
readstorage
Implementation complexitymedium
Operational complexitymedium
Scaling ceilingScan throughput scales with available I/O bandwidth and core count for vectorized execution, largely independent of table width for column-selective queries. The practical ceiling is on the write side: insert throughput is bounded by how well writes can be batched, since background merge or compaction of parts/micro-partitions cannot indefinitely outpace a high rate of small, unbatched writes.

Technologies

Canonical

clickhousesnowflake

Alternatives

trinotimescaledb

Relationships

Complements

time series rollupmaterialized view

Basis

Column-wise compression and vectorized scan advantages, and the corresponding point-write cost, are well documented in ClickHouse and Snowflake official documentation and in DDIA's column-oriented storage chapter; confidence is held slightly below a single-engine pattern because the exact mechanism (parts vs. micro-partitions) differs across the two canonical technologies.

Sources & Claims

Column-wise storage places all values for one column contiguously across rows, so a query touching a subset of a table's columns reads only that subset's data rather than every column of every matching row.

pending

ddia or accepted reference · Kleppmann, Designing Data-Intensive Applications, Chapter 3, Column-Oriented Storage

storage-engine-internals-spine batch 4

ClickHouse's MergeTree engine batches inserts into immutable parts merged asynchronously in the background; sending many single-row inserts causes excessive part fragmentation and degraded read performance.

pending

official documentation · ClickHouse documentation: MergeTree engine family

storage-engine-internals-spine batch 4

ClickHouse UPDATE and DELETE are implemented as asynchronous mutations that rewrite whole parts rather than modifying rows in place.

pending

official documentation · ClickHouse documentation: Mutations

storage-engine-internals-spine batch 4

Snowflake organizes table data into immutable micro-partitions with per-column min/max metadata that the query optimizer uses to prune micro-partitions that cannot satisfy a predicate without reading them.

pending

official documentation · Snowflake documentation: Micro-partitions and data clustering

storage-engine-internals-spine batch 4

ClickHouse documents columnar LZ4/ZSTD compression achieving roughly 5-10x reduction versus raw data size.

pending

vendor engineering post · ClickHouse documentation and engineering blog on compression codecs

storage-engine-internals-spine batch 4; carried forward from existing clickhouse.yaml technology profile figure