TLDR Tech / Data
Automated Schema Evolution in Pinterest's Next-Generation DB Ingestion Framework
Pinterest built schema evolution for CDC across Kafka, Flink, Spark, and Iceberg, treating schema as a contract. Source schemas and sink mappings generate Flink/Spark/Iceberg artifacts, while push- and pull-based checks detect drift. Changes roll out with PR auditability, SLA-based recovery, and backfill fallbacks.
Read the full story