TLDR Tech / Dev
What we learned from a 22-Day storage bug (and how we fixed it)
Mux experienced a 22-day storage bug between January 8 and February 4, during which 0.33% of audio and video segments were served in a corrupted state. An optimization to use fewer, larger storage nodes created bottlenecks that amplified pre-existing issues like remote read cancellations, a race condition in file deletions, and a bug where partially written segments were treated as complete. To fix this, Mux addressed the deletion purge path, resolved remote read context cancellation, and increased storage node counts, then regenerated all affected segments and purged CDN caches.
Read the full story