TLDR Tech / Data
Data-to-Production: Bridging the Gap Between Iceberg and Live Microservices
Wix Engineering built a self-service Data-to-Production platform to bridge the gap between their petabyte-scale Apache Iceberg data warehouse on S3 and live microservices, ingesting billions of rows into ClickHouse via dynamic, reliable Airflow-orchestrated pipelines (with strategies like atomic overwrites, upserts using ReplacingMergeTree, and partition replacements) while enforcing governance through a metadata console and human-in-the-loop reviews.
Read the full story