TLDR Tech / Data
Accelerating Apache Spark Queries (and Iceberg Rust Development) with Apache DataFusion Comet
DataFusion Comet speeds Spark reads on Iceberg by keeping Java planning while using Iceberg Rust for native Arrow execution. On a 3 TB TPC-DS benchmark, it accelerated 102 of 103 queries and cut total runtime by about 40%, while its use of Iceberg Java's Spark test suites has helped drive more than 40 Iceberg Rust pull requests.
Read the full story