↓ Skip to main content

“A 42 kB Parquet file can contain over 4 PB of data.”

Hannes Mühleisenduckdb.org ↗

One dictionary entry, a run length of 2^31-1, 1,000 copies of the data page and 290 row groups pointing at the same bytes. That gets 622 trillion values into a valid 42 kB file. Nothing in the format stops it. Every data pipeline that accepts Parquet uploads from strangers just inherited a denial-of-service bug.