TLDR Tech / Data
When 36,000 Tiny Files Break Your Spark Pipeline: A Deep Dive into S3 DNS Exhaustion and the Small File Problem
Thousands of tiny Parquet files on S3 can break Spark reads with UnknownHostException, even when networking is fine, by overwhelming DNS, S3 LIST/GET calls, and driver/task metadata. Spark partition tuning can help stabilize reads, but the real fix is compaction and table formats like Delta Lake or Iceberg.
Read the full story