TLDR Tech / Data
Writing Custom Table Providers in Apache DataFusion
DataFusion table providers let custom sources expose data from files, APIs, or proprietary systems by separating planning from execution. TableProvider::scan() runs during planning and should stay lightweight, while ExecutionPlan::execute() creates per-partition streams and SendableRecordBatchStream does the actual data work. Correctly declaring partitioning, ordering, and filter pushdown can eliminate RepartitionExec, SortExec, and wasted I/O.
Read the full story