TLDR Tech / Data
Indexing the Data Lake for Online Point Queries
Spotify built an external index mapping keys directly to file offsets and row numbers in unmodified Parquet files, replacing distributed SQL scan chains with parallel ranged reads for lookups across petabyte-scale data lakes. Complementary write-time optimizations (per-key page flushing, ZSTD frame resets, column interleaving, covering index entries) reduce both bytes read and round trips further, making the same files serve both batch analytics and interactive queries.
Read the full story