TLDR Tech / Data
Leveraging Data Assets features in Airflow 3.0 to optimise resource utilization by more than 30%
Halodoc migrated Airflow workloads from polling sensors and synchronous Redshift loads to Airflow 3.x Assets and deferrable operators. Across 160 DAGs, the team reduced worker CPU from 26.1% to 7.71% and worker memory from 49.2% to 30.8%, while also lowering scheduler load and reducing Redshift table-locking errors by about 38%.
Read the full story