TLDR Tech / Practical
Parquet file reading boosted by bitmap optimization
Apache Arrow C++ and PyArrow's next version will accelerate whole Parquet file reads by as much as 20%. This improvement comes from a new decoding method for optional columns that are not nested. Instead of creating intermediate levels, the code directly copies definition levels into Arrow's validity bitmap, avoiding extra bit-unpacking and run-length decoding. Tests demonstrate this enhancement provides substantial speed improvements, especially for columns with very few or very many null values.
Read the full story