TLDR Tech / Data
Algorithms on billion-scale graph using 10GB RAM: I love DataFusion!
Apache DataFusion can run billion-edge graph algorithms on a laptop-class memory budget by pushing graph processing into disk-backed bulk scans, sort-merge joins, aggregations, and spill-aware execution. PageRank on Graphalytics graph500-26 with 1.05B edges computes using 5GB RAM in ~30 minutes for 15 iterations, and computes weakly connected components on twitter_mpi with 1.96B edges using 10GB RAM.
Read the full story