TLDR Tech / Data
Database Federation: Decentralized and ACL-Compliant Hive Databases
Uber Engineering decentralized its monolithic Hive data warehouse by implementing Database Federation, migrating datasets into smaller, domain or team-owned Hive databases without data duplication or downtime through a one-time Bootstrap Migrator to copy data to new HDFS locations and update pointers, while real-time (Apache Flink + Piper) and batch synchronizers maintained HMS metadata consistency bidirectionally.
Read the full story