TLDR Tech / Data
The 58-Million-Key Freeze: What a HashMap Resize Taught Us About Memory Allocation at Scale
LinkedIn experienced a production incident where its Rust-based FishDB service would completely freeze for 10-15 seconds, breaching availability SLOs. The root cause was a standard library HashMap resizing at exactly 58,720,256 keys, which triggered a massive memory allocation via mmap. This acquired the process-wide mmap_lock in write mode, blocking all other threads on madvise and page faults, freezing the entire async runtime.
Read the full story