TLDR Tech / Dev
10 trillion samples a day: Scaling beyond traditional monitoring infra at Databricks
Databricks rearchitected its monitoring infrastructure to handle over 10 trillion samples daily and five billion active timeseries by transitioning to a customized timeseries database called Pantheon, which has self-healing and efficient tiered storage. This new architecture also uses an advanced aggregation pipeline to manage metric cardinality and a Lakehouse-native platform named Hydra for high-cardinality troubleshooting and 50 times cheaper data storage.
Read the full story