TLDR Tech / Practical
Zalando Engineers Tackle Circuit Breaker Issues
Zalando engineers are examining a persistent problem with client circuit breakers failing to reset. This occurs because the Horizontal Pod Autoscaler interacts with the breaker. When the circuit breaker trips, traffic decreases, causing the autoscaler to reduce the service to its minimum number of replicas. If the circuit breaker then tries to close with minimal traffic, the service passes, but the subsequent full traffic load overwhelms the scaled-down pods, leading to timeouts and re-opening the breaker. The team developed a custom tool to visualize this behavior and are considering solutions like queue-based scaling and pre-scaling.
Read the full story