TLDR Tech / DevOps
Ingesting the Milky Way: Petabyte-Scale with Zerobus Ingest
Databricks announced general availability of Zerobus Ingest, a serverless streaming service that demonstrated the ability to ingest 1 petabyte of data in under 24 hours—sustaining 12 GB/s throughput to a single Delta table—using NASA's 200 billion data point NEOWISE dataset as a benchmark. The service eliminates the need for Kafka infrastructure by using a push-based API with stream-level ordering guarantees and a custom zero-copy protobuf parser that achieves ~1 GB/s parsing per CPU core.
Read the full story