TLDR Tech / Data
Asynchronous I/O in DuckDB: Work, Thread, Work
DuckDB is adding asynchronous I/O so worker threads stop blocking on S3-backed Parquet and CSV reads. The design separates async and worker pools, adds read-ahead and memory governance, and shows some large performance wins: TPC-H Q6 on 22 GB Parquet fell from 8.23s to 2.84s, while a large CSV scan time improved by nearly 20x.
Read the full story