TLDR Tech / Data
Enabling Data Intelligence: Data Profiling Framework at Halodoc
Halodoc built an Airflow-native data profiling framework to replace repeated ad hoc SQL profiling across hundreds of tables and multiple systems. It combines column-level profiling, join intelligence, and source-table analysis, running compute in Redshift or Athena and isolating each table in Kubernetes pods with run_id-based, idempotent staging writes. The result is a self-serve, searchable view of data quality and table relationships.
Read the full story