gallery unit · data core · bottleneck class E
Columnar scan / bytes-read guardrail
“A columnar database … allows having many columns in a table just in case, but to not pay the cost for unused columns on read query execution time (a traditional OLTP database reads all of the data during queries as the data is stored in rows and not columns).”
ClickHouse documentation · “What is a columnar database?” · updated 2026-05-22
The query returns the right rows. It just read the whole table to get two columns — and nothing that checks correctness can see it.
Same file, one query, read two ways
200k rows · 2 of 12 columns · guardrail integer-exact
| Check | Full scan | Projected | |
|---|---|---|---|
| Bytes-read guardrail | FLAGGED | passes | catches it |
| Result-correctness check | right rows | right rows | misses it |
| Bytes physically read | 8.6 MB | 192 KB | 44.7x more |
Both reads return the identical 18 groups. The full scan pulled 8.6 MB through the reader to do it; the projected read pulled 192 KB — 44.7x less, measured on a real Parquet file. Against a row-oriented CSV of the same data it is 460x.
run it yourself
The same 200,000-row, 12-column table, live in your browser. Pick a query and a storage layout, and watch the guardrail read the bytes, not the rows — the correctness check only sees that the answer is right.
needed by the query · read but not needed · skipped
projection needs 800 KB · this read is 1.0x that
the query needs every column, so a full read IS minimal — the guardrail flags waste, not scans.
the answer is identical whichever layout you pick — so this check passes even the full scan.
Bytes here are the deterministic model. On a real Parquet file the Python harness measured the same effect for real: the projected read touched 193 KB versus 8.6 MB for the full scan of the same file — 44.7x — and 459.8x against a row-oriented CSV.
Why the usual check fails
- Correctness checks watch the output. “Did the query return the right rows?” is true whether the engine read two columns or all twelve. The waste is entirely on the read side, where the result never looks.
- The cost is invisible until it scales. On a laptop file it is a warm-cache blip; on a wide table in object storage it is the bill and the latency. A row layout — or a columnar file read without projection pushdown — reads every column to answer a two-column question.
The guardrail ignores the rows and asserts a property of the read: the bytes touched must be within tolerance of what the projection needs. It flags the scan that a correctness check waves through.
query: SELECT country, device ... (2 of 12 columns) row layout : read every column → 88.5 MB ✗ (460x) columnar, full read : read every chunk → 8.6 MB ✗ ( 45x) columnar, projected : read 2 column chunks → 192 KB ✓ guardrail: flag when bytes_read > tolerance x projection_budget
Evidence
Tier 4 — a real off-the-shelf control. The harness writes a real Parquet file and a real CSV, then counts the bytes physically pulled through the reader (pre-buffering off) for the same query three ways. The instrumented projected read (192 KB) is cross-checked against the file’s own column-chunk metadata (130 KB floor), so the number is I/O, not buffer coalescing. Synthetic table (read-amplification needs a controlled row width); stated on the page, not hidden.
make setup && make test && make run # $0, laptop, no GPU, no network
Honest gaps
The dataset is synthetic and deterministic (seed=7) — showing read amplification needs a controlled row width, so the numbers are on a designed table, but the failure mode is real and cited. The 44.7x is projection isolated — same file, same compression on both sides — and reflects projecting two low-cardinality dict-encoded columns; project a fat string column like url and the ratio shrinks. The 460x CSV figure additionally includes compression, so it is a second axis, not a like-for-like projection number. The guardrail covers projection (columns read); predicate pushdown / row-group skipping is a second mechanism with its own failure modes and is deliberately out of scope. The live browser demo uses the integer read model — the real compression and encoding effects live only in the measured Parquet numbers, never in the model, so Python and the TypeScript port agree to the byte.